Distributed data processing method, device, electronic device and storage medium

Abstract

The present disclosure relates to a distributed data processing method, device, electronic device and storage medium, and relates to the field of data processing technology. In this method, a driving node obtains multiple data to be processed, and distributes the multiple data to be processed to multiple processing nodes, so that each processing node iteratively processes the distributed data to be processed, and obtains a first processing result containing the results of each round of iterative processing, which can achieve higher data processing efficiency. When the number of remaining data to be processed on each processing node meets a preset condition, the driving node obtains the remaining data to be processed on each processing node, and performs one-time parallel processing on each remaining data to be processed to obtain a second processing result, which can save data processing resources and achieve higher data processing efficiency. Finally, according to the first processing result and the second processing result obtained by each processing node, the target processing result for multiple data to be processed is obtained.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a distributed data processing method, device, electronic device and storage medium. Background Art

[0002] With the continuous development of mobile Internet technology, massive amounts of data are generated every day. For large amounts of data to be processed, distributed iterative processing is usually used to process the data to obtain the expected data processing results.

[0003] However, since the duration of each iteration is relatively fixed, and after multiple rounds of iterative processing, the amount of remaining data to be processed is small, continuing to process the remaining data to be processed through iterative processing has the problems of high processing cost and low efficiency. Summary of the invention

[0004] The present disclosure provides a distributed data processing method, device, electronic device and storage medium, which can save costs and improve processing efficiency in a distributed data processing scenario.

[0005] The technical solution of the embodiment of the present disclosure is as follows:

[0006] According to a first aspect of an embodiment of the present disclosure, a distributed data processing method is provided, which is applied to a driving node in a distributed data processing system, and the distributed data processing system also includes multiple processing nodes. The distributed data processing method includes: obtaining multiple data to be processed. Distributing the multiple data to be processed to multiple processing nodes, so that each processing node iteratively processes the distributed data to be processed, and obtains a first processing result containing the results of each round of iterative processing. When the number of remaining data to be processed on each processing node meets a preset condition, the remaining data to be processed on each processing node is obtained. Each remaining data to be processed is processed in parallel to obtain a second processing result. According to the first processing result and the second processing result obtained by each processing node, a target processing result for the multiple data to be processed is obtained.

[0007] Optionally, the preset condition is that the sum of the number of remaining data to be processed on each processing node is less than a preset threshold. A specific implementation method of the preset condition of the number of remaining data to be processed on each processing node is provided. By setting the preset condition, it is possible to determine whether the sum of the number of remaining data to be processed on each processing node is less than the preset threshold, and determine the prerequisite for obtaining the remaining data to be processed on each processing node.

[0008] Optionally, the preset condition is that the number of remaining data to be processed on each processing node is less than a preset threshold. A preset condition for the number of remaining data to be processed on each processing node is provided, and the preset threshold is determined by two factors: the preset memory capacity of the driving node and the preset number of remaining data to be processed on each processing node. When the number of data to be processed is large, it can be processed iteratively, and when the number of data to be processed is small, it can be processed in parallel, so as to effectively save processing resources, reduce processing time, and improve processing efficiency.

[0009] Optionally, the preset threshold is determined based on the preset memory capacity of the driving node and the preset amount of remaining data of each processing node, wherein the preset amount of remaining data of each processing node refers to the amount of remaining data to be processed when the convergence speed of the historical iterative processing process executed by the processing node is less than the preset convergence speed after the target round.

[0010] Optionally, when the amount of remaining data to be processed on each processing node meets a preset condition, obtaining the remaining data to be processed on each processing node includes: when the amount of remaining data to be processed on each processing node meets the preset condition, determining the first memory capacity required for parallel processing of the remaining data to be processed according to the amount of remaining data to be processed on each processing node. When the first memory capacity is less than the second memory capacity, obtaining the remaining data to be processed on each processing node, the second memory capacity being the current free memory capacity of the driver node.

[0011] Optionally, the distributed data processing system further includes a configuration node. When the amount of remaining data to be processed on each processing node meets a preset condition, the remaining data to be processed on each processing node is obtained, and further includes: when the first memory capacity is greater than or equal to the second memory capacity, sending a memory adjustment request to the configuration node to request the configuration node to adjust the second memory capacity so that the first memory capacity is less than the second memory capacity. When the configuration node completes adjusting the second memory capacity, the remaining data to be processed on each processing node is obtained.

[0012] Optionally, each remaining data to be processed is processed in parallel to obtain a second processing result, including: converting the remaining data to be processed in a first format to obtain the remaining data to be processed in a second format, wherein the first format refers to a data format that matches iterative processing, and the second format refers to a data format that matches parallel processing. The remaining data to be processed in the second format are processed in parallel to obtain a second processing result. By converting the format of the data to be processed from each processing node into a format suitable for parallel processing, it is possible to facilitate the processing of the driver node and improve processing efficiency.

[0013] According to a second aspect of an embodiment of the present disclosure, a distributed data processing device is provided, which is applied to a driving node in a distributed data processing system, and the distributed data processing system also includes multiple processing nodes. The distributed data processing device includes: a first data acquisition unit, configured to execute acquisition of multiple data to be processed. A data distribution unit, configured to execute distribution of multiple data to be processed to multiple processing nodes, so that each processing node performs iterative processing on the distributed data to be processed, and obtains a first processing result containing the results of each round of iterative processing. A second data acquisition unit, configured to execute when the number of remaining data to be processed on each processing node meets a preset condition, to obtain the remaining data to be processed on each processing node. The data processing unit, configured to perform parallel processing on each remaining data to be processed, and obtain a second processing result. A result determination unit, configured to execute the first processing result and the second processing result obtained according to each processing node, to obtain a target processing result for multiple data to be processed.

[0014] Optionally, the preset condition is that the sum of the amount of remaining data to be processed on each processing node is less than a preset threshold.

[0015] Optionally, the preset condition is that the amount of remaining data to be processed on each processing node is less than a preset threshold.

[0016] Optionally, the preset threshold is determined based on the preset memory capacity of the driving node and the preset amount of remaining data of each processing node, wherein the preset amount of remaining data of each processing node refers to the amount of remaining data to be processed when the convergence speed of the historical iterative processing process executed by the processing node is less than the preset convergence speed after the target round.

[0017] Optionally, the second data acquisition unit is specifically configured to execute: when the amount of remaining data to be processed on each processing node meets a preset condition, determine the second memory capacity required for parallel processing of the remaining data to be processed according to the amount of remaining data to be processed on each processing node. When the first memory capacity is less than the second memory capacity, acquire the remaining data to be processed on each processing node, and the second memory capacity is the current free memory capacity of the driver node.

[0018] Optionally, the distributed data processing system further includes a configuration node. The second data acquisition unit is further configured to execute: when the first memory capacity is greater than or equal to the second memory capacity, send a memory adjustment request to the configuration node to request the configuration node to adjust the second memory capacity so that the first memory capacity is less than the second memory capacity. The second data acquisition unit is further configured to execute when the configuration node completes adjusting the second memory capacity, obtain the remaining to-be-processed data on each processing node.

[0019] Optionally, the result determination unit is specifically configured to perform: converting the remaining data to be processed in a first format to obtain the remaining data to be processed in a second format, wherein the first format refers to a data format that matches the iterative processing, and the second format refers to a data format that matches the parallel processing. Parallel processing is performed on the remaining data to be processed in the second format to obtain a second processing result.

[0020] According to a third aspect of the embodiments of the present disclosure, an electronic device is provided, which may include: a processor and a memory for storing instructions executable by the processor, wherein the processor is configured to execute the instructions to implement any one of the optional distributed data processing methods in the first aspect above.

[0021] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which instructions are stored. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute any one of the optional distributed data processing methods in the above-mentioned first aspect.

[0022] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, which, when the computer program / instructions are executed by a processor, implements a distributed data processing method as optionally implemented in any one of the first aspects.

[0023] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

[0024] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0025] Based on any of the above aspects of the present disclosure, a distributed data processing method is provided, comprising: obtaining a plurality of data to be processed. Distributing the plurality of data to be processed to a plurality of processing nodes, so that each processing node performs iterative processing on the distributed data to be processed, and obtains a first processing result including the result of each round of iterative processing. When the number of remaining data to be processed on each processing node meets a preset condition, the remaining data to be processed on each processing node is obtained. Each remaining data to be processed is processed in parallel to obtain a second processing result. According to the first processing result and the second processing result obtained by each processing node, a target processing result for the plurality of data to be processed is obtained. Since the number of remaining data to be processed is small after the processing node performs multiple rounds of iterative processing on the data to be processed, it still needs to be divided into multiple dimensions for multiple rounds of iterative processing, and the time required for each round of iterative processing is relatively fixed, so there are problems of long processing time and low processing efficiency. At the same time, since the number of remaining data to be processed in each round of processing is small, a large number of processing resources are idle, resulting in a waste of processing resources. Based on this, the present disclosure processes the remaining data to be processed in parallel at one time through the driver node when the amount of the remaining data to be processed meets the preset conditions, which can save processing resources and improve processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0027] Figure 1 A schematic diagram of the structure of a communication system provided by an embodiment of the present disclosure is shown;

[0028] Figure 2 A schematic diagram of the structure of a distributed data processing system provided by an embodiment of the present disclosure is shown;

[0029] Figure 3 A schematic diagram of a distributed data processing method provided by an embodiment of the present disclosure is shown;

[0030] Figure 4 A schematic diagram of a distributed data processing method provided by an embodiment of the present disclosure is shown;

[0031] Figure 5 A schematic diagram of a distributed data processing method provided by an embodiment of the present disclosure is shown;

[0032] Figure 6 A schematic diagram of a distributed data processing method provided by an embodiment of the present disclosure is shown;

[0033] Figure 7A schematic diagram of the structure of a distributed data processing device provided by an embodiment of the present disclosure is shown;

[0034] Figure 8 A schematic structural diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0035] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0036] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0037] It should also be understood that the term “comprising” indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements and / or components.

[0038] It should be noted that the user information (including but not limited to user device information, user personal information, user behavior information, etc.) and data (including but not limited to page data corresponding to dynamic pages, etc.) involved in this disclosure are all data authorized by the user or fully authorized by all parties.

[0039] With the continuous development of mobile Internet technology, massive amounts of data are generated every day. For large amounts of data to be processed, distributed iterative processing is usually used to process the data to obtain the expected data processing results.

[0040] In the related art, a distributed data processing system based on Spark (hereinafter referred to as a distributed data processing system) is used to process the above-mentioned data to be processed. The distributed data processing system includes a driver node, multiple processing nodes and a configuration node. When the driver node receives a Spark job including multiple data to be processed, it generates a corresponding Driver process according to the Spark job. The Driver process is used to request the configuration node to process the resources required for the above-mentioned Spark job. The configuration node starts a certain number of Executor processes on multiple distributed processing nodes according to the request, wherein the configuration node can be a cluster manager. Then, the driver node splits the Spark job through the Driver process and distributes it to each processing node that starts the Executor process, and the processing node processes the data to be processed through the Executor process. For each processing node, data is usually processed by iterative processing. Each round of iteration processes the data to be processed that meets the preset conditions of the current round respectively, and the data to be processed that does not meet the preset conditions of the current round will enter the next round until all the data to be processed are processed.

[0041] However, since the duration of each round of iteration is relatively fixed, and after multiple rounds of iterative processing, the amount of data remaining to be processed is relatively small, continuing to process the remaining data to be processed by iterative processing has the problem of high processing cost and low efficiency. In particular, in some scenarios, the data to be processed with a high order of magnitude is distributed in a long-tailed skewed manner, that is, the data to be processed belongs to a large number of different dimensions, but the amount of data in each dimension is small. In this way, even if the amount of data to be processed is small, multiple rounds of iterative calculations are still required, resulting in a waste of data processing resources, a long processing time, and a serious impact on processing efficiency.

[0042] In the related technology, the problem of too long execution plan is usually avoided through Spark's checkpoint mechanism, but it involves modifying the system code, which has a certain performance overhead. At the same time, the data to be processed is intercepted in advance to reduce the amount of remaining data to be processed after several rounds of iterative processing. This method will cause the loss of part of the data to be processed, resulting in reduced accuracy of the data processing results.

[0043] Based on this, an embodiment of the present disclosure provides a distributed data processing method, including: obtaining multiple data to be processed. Distributing the multiple data to be processed to multiple processing nodes, so that each processing node iteratively processes the distributed data to be processed, and obtains a first processing result including the result of each round of iterative processing. When the number of remaining data to be processed on each processing node meets a preset condition, the remaining data to be processed on each processing node is obtained. Each remaining data to be processed is processed in parallel to obtain a second processing result. According to the first processing result and the second processing result obtained by each processing node, a target processing result for the multiple data to be processed is obtained.

[0044] After the processing node performs multiple rounds of iterative processing on the data to be processed, although the amount of the remaining data to be processed is small, it still needs to be divided into multiple dimensions and multiple rounds of iterative processing are performed, and the time required for each round of iterative processing is relatively fixed, so there are problems of long processing time and low processing efficiency. At the same time, since the amount of remaining data to be processed in each round of processing is small, most of the resources on the processing node are idle, resulting in a waste of processing resources. Based on this, the present disclosure, when the amount of remaining data to be processed meets the preset conditions, the remaining data to be processed is processed in parallel once through the driving node, which can save processing resources and improve processing efficiency.

[0045] The following is an exemplary description of the application scenario of the distributed data processing method provided by the embodiment of the present disclosure:

[0046] Figure 1 A communication system schematic diagram provided in an embodiment of the present disclosure is shown in FIG. Figure 1 As shown, the communication system may include: a server 110, a terminal device 120 and a distributed data processing system 130. The server 110 may establish a connection with the terminal device 120 and the distributed data processing system 130 via a wired network or a wireless network.

[0047] The server 110 can be used to provide services for the applications installed on the terminal device 120 so that the applications on the terminal device 120 can realize corresponding functions. At the same time, the server 110 is also used to obtain a large amount of data generated by the terminal device during the operation of the application, and send the generated data as data to be processed to the distributed data processing system 130 for processing. In some embodiments, the server 110 is a distributed cluster composed of multiple servers (or micro servers).

[0048] The terminal device 120 can be used to load different applications and store a large amount of data to be processed during the operation of the application. The data to be processed is sent to the server 110. The data to be processed can be used to obtain the processing result of the terminal device 120 after being processed by the distributed data processing system 130.

[0049] In some embodiments, the terminal device may be a mobile phone, tablet computer, desktop, laptop, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, cellular phone, personal digital assistant (PDA), augmented reality (AR)\virtual reality (VR) device, etc., which can install and use various applications (such as Kuaishou), and the present disclosure does not impose any special restrictions on the specific form of the terminal. It can interact with the user through one or more methods such as keyboard, touchpad, touch screen, remote control, voice interaction or handwriting device.

[0050] The distributed data processing system 130 is used to receive the data to be processed from the server 110 , process the data to be processed, obtain corresponding processing results, and return the processing results to the server 110 .

[0051] Optionally, the above Figure 1 In the communication system shown, the server 110 may be connected to at least one terminal device 120. The present disclosure does not limit the number and type of the terminal devices 120.

[0052] The distributed data processing method provided by the embodiment of the present disclosure can be applied to the aforementioned Figure 1 The distributed data processing system 130 shown can also be applied to other electronic devices.

[0053] In other implementations, the executor of the distributed data processing method provided by the present disclosure may be a distributed data processing device, which may be built into an electronic device or may be an independent device.

[0054] Figure 2 A schematic diagram of a distributed data processing system exemplarily shown in the present disclosure is shown in FIG. Figure 2 As shown, the distributed data processing system 130 includes a configuration driver node 131 , a configuration node 132 and multiple processing nodes 133 .

[0055] Among them, the driver node 131 is used to obtain multiple data to be processed, and request the configuration node 132 to process the processing resources required for the above-mentioned data to be processed according to the multiple data to be processed, so that the configuration node 132 determines multiple processing nodes 133 and allocates resources to multiple processing nodes 133. The driver node 131 is also used to split the above-mentioned multiple data to be processed, and send the split data to be processed to multiple processing nodes 133. Each processing node performs multiple rounds of iterative processing on the distributed data to be processed to obtain a first processing result containing the processing results of each round. In the case where the above-mentioned distributed data processing system is a distributed framework based on Spark, the driver node 131 is specifically a node running the Driver process.

[0056] The driver node 131 is further used to retrieve the remaining data to be processed in each processing node 133 when the amount of remaining data to be processed on the processing node 133 meets a preset threshold, perform one-time parallel processing, and obtain a corresponding second processing result.

[0057] The configuration node 132 is used to determine multiple processing nodes 133 according to the request sent by the driver node 131, so as to perform distributed processing on the above-mentioned multiple data to be processed. In the case where the above-mentioned distributed data processing system is a distributed framework based on Spark, the configuration node 132 is specifically a cluster manager. Each processing node 133 is used to receive the data to be processed distributed by the driver node 131, and iteratively process the data to be processed to obtain corresponding processing results. It is also used to return the remaining data to be processed to the driver node 131 when the number of remaining data to be processed meets the preset conditions. In the case where the above-mentioned distributed data processing system is a distributed framework based on Spark, the processing node 133 is used to run the Executor process.

[0058] In some implementations, the distributed data processing system 130 further includes a storage node 134 for storing the first processing result obtained by the processing node 133 and the second processing result obtained by the driver node 131. In one implementation, the storage node 134 performs storage using a distributed file storage technology.

[0059] Figure 3 A flow chart of a distributed data processing method provided by an embodiment of the present disclosure is as follows: Figure 3 As shown, when the distributed data processing method is applied to the above-mentioned driver node 131, the distributed data processing method may include:

[0060] S301, obtaining a plurality of data to be processed.

[0061] The above-mentioned data to be processed is data generated when the user object uses the target application on the terminal device. The multiple data to be processed include data generated when logging into the target application with different user accounts on the same terminal device, and may also include data generated when logging into the target application with the same user account on different terminal devices.

[0062] S302: Distribute a plurality of data to be processed to a plurality of processing nodes, so that each processing node iteratively processes the distributed data to be processed to obtain a first processing result including the result of each round of iterative processing.

[0063] In one implementation, multiple data to be processed are split, and the split multiple groups of data to be processed are sent to different processing nodes respectively, and the processing nodes perform iterative processing on the received data to be processed. Among them, iterative processing can be understood as performing multiple rounds of processing on the data to be processed, each round corresponds to different preset conditions, and when the data to be processed meets the preset conditions of the current round, it is processed in the current round. When the data to be processed does not meet the preset conditions of the current round, it will be processed in subsequent rounds. When the amount of data to be processed is large, iterative processing can effectively improve processing efficiency and save processing time.

[0064] S303: When the amount of remaining data to be processed on each processing node meets a preset condition, the remaining data to be processed on each processing node is obtained.

[0065] In one implementation, the Spark framework includes a preset method for determining the amount of data and a method for obtaining the data to be processed. For example, the preset method for determining the amount of data may be a count method, and the method for obtaining the data to be processed may be a collect method. The driver node determines the amount of remaining data to be processed of each processing node after each round of iterative processing by calling the count method. When the amount of remaining data to be processed meets the preset condition, the driver node collects the remaining data to be processed in each processing node to the driver node by calling the collect method.

[0066] S304: Process the remaining data to be processed in parallel to obtain a second processing result.

[0067] S305 . Obtain target processing results for a plurality of to-be-processed data according to the first processing results and the second processing results obtained by each processing node.

[0068] After the processing node performs multiple rounds of iterative processing on the data to be processed, although the amount of the remaining data to be processed is small, it still needs to be divided into multiple dimensions and multiple rounds of iterative processing are performed, and the time required for each round of iterative processing is relatively fixed, so there are problems of long processing time and low processing efficiency. At the same time, since the amount of remaining data to be processed in each round of processing is small, most of the resources on the processing node are idle, resulting in a waste of processing resources. Based on this, the present disclosure, when the amount of remaining data to be processed meets the preset conditions, the remaining data to be processed is processed in parallel once through the driving node, which can save processing resources and improve processing efficiency.

[0069] In some implementations, the preset condition is that the sum of the amounts of remaining data to be processed on each processing node is less than a preset threshold.

[0070] For example, after 5 rounds of iterative processing, the amount of remaining data to be processed by processing node A is 200, after 5 rounds of iterative processing by processing node B, the amount of remaining data to be processed is 100, and after 5 rounds of iterative processing by processing node C, the amount of remaining data to be processed is 300. When the preset threshold is 1000, the sum of the amount of remaining data to be processed of the three processing nodes is 600, which is less than the preset threshold 1000. At this time, the amount of remaining data to be processed on each processing node meets the preset condition.

[0071] From the above, it can be seen that a specific implementation method is provided for limiting the number of remaining data to be processed of each processing node. By setting a preset condition, when the sum of the number of remaining data to be processed of each processing node is less than a preset threshold, the driver retrieves the remaining data to be processed of each processing node and performs one-time parallel processing, which effectively saves data processing resources and improves processing efficiency.

[0072] In some other implementations, the preset condition is that the amount of remaining data to be processed on each processing node is less than a preset threshold.

[0073] For example, after 5 rounds of iterative processing on processing node A, the amount of remaining data to be processed is 200; after 5 rounds of iterative processing on processing node B, the amount of remaining data to be processed is 100; after 5 rounds of iterative processing on processing node C, the amount of remaining data to be processed is 300. When the preset threshold is 400, the amount of remaining data to be processed on the three processing nodes is less than the preset threshold of 400. At this time, the amount of remaining data to be processed on each processing node meets the preset conditions.

[0074] From the above, it can be seen that a specific implementation method is provided for limiting the number of remaining data to be processed of each processing node. By setting a preset condition, when the number of remaining data to be processed of each processing node is less than a preset threshold, the driver retrieves the remaining data to be processed of each processing node and performs one-time parallel processing, which effectively saves data processing resources and improves processing efficiency.

[0075] In some implementations, the preset threshold is determined based on the preset memory capacity of the driving node and the preset amount of remaining data of each processing node, wherein the preset amount of remaining data of each processing node refers to the amount of remaining data to be processed when the convergence speed of the historical iterative processing process executed by the processing node is less than the preset convergence speed after the target round.

[0076] In the above implementation, the preset memory capacity of the driver node and the number of remaining data to be processed preset by each processing node are determined because: on the one hand, since the memory capacity of the driver node is limited, if the preset threshold is set too high, the number of remaining data to be processed is too high, resulting in a memory overflow problem in the driver node. On the other hand, since parallel processing cannot reduce the processing time when the magnitude of the data to be processed is large, it is more effective to save processing time and improve processing efficiency for data to be processed with a large amount of data through iterative processing. When the magnitude of the data to be processed is small, it still requires multiple rounds of iterative processing, and the processing time of each round is relatively fixed. At this time, by processing the smaller magnitude of the data to be processed in parallel, resources can be effectively saved and processing efficiency can be improved.

[0077] From the above, it can be seen that a specific implementation method for determining a preset threshold is provided. The preset threshold is determined by two factors: the memory capacity preset by the driving node and the number of remaining data to be processed preset by each processing node. When the amount of data to be processed is large, it can be processed iteratively when the amount of data to be processed is large, and when the amount of data to be processed is small, it can be processed in parallel, so as to effectively save processing resources, reduce processing time and improve processing efficiency.

[0078] In some implementations, see Figure 4 The above S303 specifically includes the following steps:

[0079] S401. When the amount of remaining data to be processed on each processing node meets a preset condition, determine a first memory capacity required for parallel processing of the remaining data to be processed according to the amount of remaining data to be processed on each processing node.

[0080] S402: When the first memory capacity is smaller than the second memory capacity, obtain remaining data to be processed on each processing node.

[0081] In the above implementation, processing the remaining data to be processed requires memory capacity. Before obtaining the amount of remaining data to be processed on each processing node, it is necessary to confirm the memory capacity required to process the remaining data to be processed on each processing node. When the memory capacity required for the remaining data to be processed on each processing node is less than the memory capacity pre-set by the driving node, the remaining data to be processed is obtained from each processing node.

[0082] Exemplarily, after each processing node completes a round of iterative processing, the memory capacity required for the remaining data to be processed on each processing node is 10GB, and the pre-set content capacity is 12GB. At this time, the memory capacity required for the remaining data to be processed on each processing node is less than the pre-set memory capacity of the driving node. At this time, the remaining data to be processed on each processing node is obtained.

[0083] As can be seen from the above, by determining the memory capacity required for the remaining data to be processed on each processing node, it is possible to avoid the problem of memory overflow caused by the memory capacity required for the remaining data to be processed on each processing node exceeding the preset memory capacity.

[0084] In some implementations, the distributed data processing system further includes a configuration node, see Figure 5 The above S303 further includes:

[0085] S501. When a first memory capacity is greater than or equal to a second memory capacity, send a memory adjustment request to a configuration node to request the configuration node to adjust the second memory capacity so that the first memory capacity is smaller than the second memory capacity.

[0086] S502: When the configuration node completes adjusting the second memory capacity, obtain the remaining data to be processed on each processing node.

[0087] In the above implementation, when the memory capacity required for the remaining data to be processed on each processing node is greater than the memory capacity preset by the driver node, a memory adjustment request is sent to the configuration node so that the memory capacity of the driver node is adjusted to be greater than the memory capacity required for the remaining data to be processed on each processing node, and when the adjustment is completed, the remaining data to be processed is obtained from each processing node. In the case where the above distributed data processing system is a distributed framework based on Spark, the memory capacity of the driver node is adjusted by setting the parameters of Spark.

[0088] Exemplarily, after each processing node completes a round of iterative processing, the memory capacity required for the remaining data to be processed on each processing node is 12GB, and the pre-set content capacity is 10GB. At this time, the memory capacity required for the remaining data to be processed on each processing node is greater than the pre-set memory capacity of the driving node. At this time, a request is sent to the configuration node. After the configuration node adjusts the memory capacity of the driving node to 12GB, the remaining data to be processed on each processing node is obtained.

[0089] From the above, it can be seen that by determining the memory capacity required by the remaining data to be processed on each processing node, when the memory capacity required by the remaining data to be processed on each processing node is greater than the pre-set memory capacity, the memory overflow problem can be avoided by adjusting the memory capacity.

[0090] In some implementations, see Figure 6 The above step S305 specifically includes:

[0091] S601. Convert the remaining data to be processed in a first format to obtain the remaining data to be processed in a second format. The first format refers to a data format that matches iterative processing, and the second format refers to a data format that matches parallel processing.

[0092] S602: Perform parallel processing on the remaining to-be-processed data in the second format to obtain a second processing result.

[0093] Exemplarily, the first format may be the Dataset[Row] format, i.e., the data to be processed is arranged in a row format, and the second format may be the Dataset[T] format, i.e., the data to be processed is arranged in a representation form. The Dataset[Row] format facilitates iterative processing by each processing node, but is not conducive to parallel processing by the driving node. Therefore, the Dataset[Row] format is a data format that matches iterative processing. The Dataset[T] format facilitates parallel processing by the driving node. Therefore, the Dataset[T] format is a data format that matches iterative processing.

[0094] As can be seen from the above, by converting the format of the data to be processed from each processing node into a format suitable for parallel processing, it is possible to facilitate the processing by the driving node and improve the processing efficiency.

[0095] In some implementations, the driver node may be the same node as the configuration node, or may be any node in the distributed system for running the Driver process, which is not limited here.

[0096] It is understandable that, in actual implementation, the terminal / server of the embodiment of the present disclosure may include one or more hardware structures and / or software modules for implementing the aforementioned corresponding distributed data processing method, and these execution hardware structures and / or software modules may constitute an electronic device. It should be easily appreciated by those skilled in the art that, in combination with the algorithm steps of each example described in the embodiments disclosed herein, the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present disclosure.

[0097] Based on such understanding, the embodiments of the present disclosure also provide a distributed data processing device that can be applied to electronic devices.

[0098] Figure 7 FIG. 2 shows a schematic diagram of the structure of a distributed data processing device provided by an embodiment of the present disclosure. Figure 7 As shown, the distributed data processing device may include: a first data acquisition unit 701, a data distribution unit 702, a second data acquisition unit 703, a data processing unit 704 and a result determination unit 705. The first data acquisition unit 701 is configured to execute the acquisition of multiple data to be processed, for example, for executing step S301 in the above method. The data distribution unit 702 is configured to execute the distribution of multiple data to be processed to multiple processing nodes, so that each processing node performs iterative processing on the distributed data to be processed, and obtains a first processing result including the result of each round of iterative processing, for example, for executing step S302 in the above method. The second data acquisition unit 703 is configured to execute when the number of remaining data to be processed on each processing node meets a preset condition, and obtains the remaining data to be processed on each processing node, for example, for executing step S303 in the above method. The data processing unit 704 is configured to execute parallel processing on each remaining data to be processed, and obtain a second processing result, for example, for executing step S304 in the above method. The result determination unit 705 is configured to execute the first processing result and the second processing result obtained according to each processing node to obtain a target processing result for a plurality of to-be-processed data, for example, to execute step S305 in the above method.

[0099] Optionally, the preset condition is that the sum of the amount of remaining data to be processed on each processing node is less than a preset threshold.

[0100] Optionally, the preset condition is that the amount of remaining data to be processed on each processing node is less than a preset threshold.

[0101] Optionally, the preset threshold is determined based on the preset memory capacity of the driving node and the preset amount of remaining data of each processing node, wherein the preset amount of remaining data of each processing node refers to the amount of remaining data to be processed when the convergence speed of the historical iterative processing process executed by the processing node is less than the preset convergence speed after the target round.

[0102] Optionally, the second data acquisition unit 703 is specifically configured to execute: when the amount of remaining data to be processed on each processing node meets a preset condition, determine the second memory capacity required for parallel processing of the remaining data to be processed according to the amount of remaining data to be processed on each processing node. When the first memory capacity is less than the second memory capacity, acquire the remaining data to be processed on each processing node, and the second memory capacity is the current free memory capacity of the driver node, for example, for executing steps S401-S402 in the above method.

[0103] Optionally, the distributed data processing system further includes a configuration node. The second data acquisition unit 703 is further configured to execute: when the first memory capacity is greater than or equal to the second memory capacity, send a memory adjustment request to the configuration node to request the configuration node to adjust the second memory capacity so that the first memory capacity is less than the second memory capacity. The second data acquisition unit is further configured to execute when the configuration node completes the adjustment of the second memory capacity, obtain the remaining data to be processed on each processing node, for example, to execute steps S501-S502 in the above method.

[0104] Optionally, the result determination unit is specifically configured to perform: converting the remaining data to be processed in a first format to obtain the remaining data to be processed in a second format, wherein the first format refers to a data format that matches iterative processing, and the second format refers to a data format that matches parallel processing. Parallel processing is performed on the remaining data to be processed in the second format to obtain a second processing result, for example, for executing steps S601-S602 in the above method.

[0105] As described above, the embodiments of the present disclosure can divide the functional modules of the electronic device according to the above method examples. Among them, the above integrated modules can be implemented in the form of hardware or in the form of software functional modules. In addition, it should be noted that the division of modules in the embodiments of the present disclosure is schematic and is only a logical function division. There may be other division methods in actual implementation. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module.

[0106] Regarding the distributed data processing device in the above embodiment, the specific manner in which each module performs operations and the beneficial effects possessed have been described in detail in the aforementioned method embodiment and will not be repeated here.

[0107] The embodiment of the present disclosure also provides an electronic device. Figure 8 The electronic device provided by the embodiment of the present disclosure is shown in the structural diagram. The electronic device may be a distributed data processing device and may include at least one processor 801 , a communication bus 802 , a memory 803 and at least one communication interface 804 .

[0108] Processor 801 may be a processor (central processing unit, CPU), microprocessor unit, ASIC, or one or more integrated circuits for controlling the execution of the program of the disclosed solution. Figure 7 The functions implemented by the first data acquisition unit 701, the data distribution unit 702, the second data acquisition unit 703, the data processing unit 704 and the result determination unit 705 in the electronic device are similar to those implemented by Figure 8 The functions implemented by the processor 801 in are the same.

[0109] Communication bus 802 may include a pathway for transmitting information between the above-mentioned components.

[0110] The communication interface 804 uses any transceiver-like device for communicating with other devices or communication networks, such as servers, Ethernet, radio access networks (RAN), wireless local area networks (WLAN), etc. As an example,

[0111] The memory 803 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory may exist independently and be connected to the processing unit through a bus. The memory may also be integrated with the processing unit.

[0112] The memory 803 is used to store application code for executing the solution of the present disclosure, and the execution is controlled by the processor 801. The processor 801 is used to execute the application code stored in the memory 803, so as to realize the functions in the method of the present disclosure.

[0113] In a specific implementation, as an embodiment, the processor 801 may include one or more CPUs, such as Figure 8 CPU0 and CPU1 in.

[0114] In a specific implementation, as an embodiment, the electronic device may include multiple processors, such as Figure 8 801 and processor 805 in the embodiment of the present invention. Each of these processors may be a single-CPU processor or a multi-CPU processor. The processor herein may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0115] In a specific implementation, as an embodiment, the electronic device may further include an input device 806 and an output device 807. The input device 806 communicates with the output device 807 and may receive user input in a variety of ways. For example, the input device 806 may be a mouse, a keyboard, a touch screen device, or a sensor device. The output device 807 communicates with the processor 801 and may display information in a variety of ways. For example, the output device 801 may be a liquid crystal display (LCD), a light emitting diode (LED) display device, etc.

[0116] Those skilled in the art will understand that Figure 8 The structure shown in the figure does not constitute a limitation on the electronic device, and may include more or less components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0117] The disclosed embodiments also provide an electronic device. The electronic device may be a distributed data processing device. The electronic device may have relatively large differences due to different configurations or performances, and may include one or more processors and one or more memories. Among them, at least one instruction is stored in the memory, and at least one instruction is loaded and executed by the processor to implement the distributed data processing method provided by each of the above method embodiments. Of course, the electronic device may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The electronic device may also include other components for implementing the functions of the device, which will not be described in detail here.

[0118] The present disclosure also provides a computer-readable storage medium including instructions, and the computer-readable storage medium stores instructions. When the instructions in the computer-readable storage medium are executed by the processor of a computer device, the computer can execute the distributed data processing method provided by the above-mentioned embodiment. For example, the computer-readable storage medium can be a memory 803 including instructions, and the above instructions can be executed by the processor 801 of the terminal to complete the above method. For another example, the computer-readable storage medium can be a memory including instructions, and the above instructions can be executed by the processor of an electronic device to complete the above method. Optionally, the computer-readable storage medium can be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium can be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0119] The present disclosure also provides a computer program product, which includes computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned distributed data processing method.

[0120] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0121] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A distributed data processing method, characterized in that: The method is applied to a driver node in a distributed data processing system, wherein the distributed data processing system further comprises a plurality of processing nodes. Get multiple data to be processed; Distributing the plurality of data to be processed to the plurality of processing nodes, so that each processing node performs iterative processing on the distributed data to be processed, and obtains a first processing result including a result of each round of iterative processing; When the amount of remaining data to be processed on each processing node meets a preset condition, the remaining data to be processed on each processing node is obtained; the preset condition is: the sum of the amounts of remaining data to be processed on each processing node is less than a preset threshold, or the amounts of remaining data to be processed on each processing node are all less than the preset threshold; the preset threshold is determined according to the preset memory capacity of the driving node and the preset amount of remaining data on each processing node, wherein the preset amount of remaining data on each processing node refers to the amount of remaining data to be processed when the convergence speed of the historical iterative processing process executed by the processing node is less than the preset convergence speed after the target round. Processing the remaining data to be processed in parallel to obtain a second processing result; A target processing result for the plurality of data to be processed is obtained according to the first processing result and the second processing result obtained by each processing node.

2. The distributed data processing method according to claim 1, characterized in that: When the amount of remaining data to be processed on each processing node meets a preset condition, obtaining the remaining data to be processed on each processing node includes: When the amount of remaining data to be processed on each of the processing nodes meets a preset condition, determining a first memory capacity required for parallel processing of the remaining data to be processed according to the amount of remaining data to be processed on each of the processing nodes; In the case where the first memory capacity is smaller than the second memory capacity, the remaining data to be processed on each of the processing nodes is obtained, and the second memory capacity is the current free memory capacity of the driver node.

3. The distributed data processing method according to claim 2, characterized in that: The distributed data processing system further includes a configuration node; when the amount of remaining data to be processed on each processing node meets a preset condition, obtaining the remaining data to be processed on each processing node, further includes: When the first memory capacity is greater than or equal to the second memory capacity, sending a memory adjustment request to the configuration node to request the configuration node to adjust the second memory capacity so that the first memory capacity is less than the second memory capacity; When the configuration node completes adjusting the second memory capacity, the remaining data to be processed on each of the processing nodes is obtained.

4. The distributed data processing method according to claim 1, characterized in that: The performing parallel processing on each of the remaining data to be processed to obtain a second processing result includes: Convert the remaining data to be processed in a first format to obtain the remaining data to be processed in a second format, wherein the first format refers to a data format matching the iterative processing, and the second format refers to a data format matching the parallel processing; The remaining to-be-processed data in the second format are processed in parallel to obtain the second processing result.

5. A distributed data processing device, characterized in that: A driver node applied to a distributed data processing system, wherein the distributed data processing system further comprises a plurality of processing nodes; the device comprises: A first data acquisition unit is configured to acquire a plurality of to-be-processed data; A data distribution unit is configured to distribute the plurality of to-be-processed data to the plurality of processing nodes, so that each processing node performs iterative processing on the distributed to-be-processed data to obtain a first processing result including a result of each round of iterative processing; The second data acquisition unit is configured to acquire the remaining data to be processed on each processing node when the amount of remaining data to be processed on each processing node meets a preset condition; the preset condition is: the sum of the amounts of remaining data to be processed on each processing node is less than a preset threshold, or the amounts of remaining data to be processed on each processing node are all less than the preset threshold; the preset threshold is determined according to the preset memory capacity of the driving node and the preset amount of remaining data on each processing node, wherein the preset amount of remaining data on each processing node refers to the amount of remaining data to be processed when the convergence speed of the historical iterative processing process executed by the processing node is less than the preset convergence speed after the target round; A data processing unit is configured to perform parallel processing on each of the remaining data to be processed to obtain a second processing result; The result determination unit is configured to execute the first processing result and the second processing result obtained according to each of the processing nodes to obtain a target processing result for the plurality of data to be processed.

6. The distributed data processing device according to claim 5, characterized in that: The second data acquisition unit is specifically configured to execute: When the amount of remaining data to be processed on each of the processing nodes meets a preset condition, determining the second memory capacity required for parallel processing of the remaining data to be processed according to the amount of remaining data to be processed on each of the processing nodes; In the case where the first memory capacity is smaller than the second memory capacity, the remaining data to be processed on each of the processing nodes is obtained, and the second memory capacity is the current free memory capacity of the driver node.

7. The distributed data processing device according to claim 6, characterized in that: The distributed data processing system further includes a configuration node; the second data acquisition unit is further configured to execute: When the first memory capacity is greater than or equal to the second memory capacity, sending a memory adjustment request to the configuration node to request the configuration node to adjust the second memory capacity so that the first memory capacity is less than the second memory capacity; The second data acquisition unit is further configured to acquire the remaining data to be processed on each of the processing nodes when the configuration node completes adjusting the second memory capacity.

8. The distributed data processing device according to claim 5, characterized in that: The result determination unit is specifically configured to execute: Convert the remaining data to be processed in a first format to obtain the remaining data to be processed in a second format, wherein the first format refers to a data format matching the iterative processing, and the second format refers to a data format matching the parallel processing; The remaining to-be-processed data in the second format are processed in parallel to obtain the second processing result.

9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the distributed data processing method according to any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the distributed data processing method as described in any one of claims 1 to 4.

11. A computer program product, characterized in that The computer program product comprises computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the distributed data processing method according to any one of claims 1 to 4.

Images