A processing method and device applied to a distributed computing engine

Abstract

Embodiments of the present application relate to the technical field of distributed computing, and particularly relate to a processing method and device applied to a distributed computing engine. The method comprises: a control node obtaining an indication value of a second accumulator corresponding to each task in each execution node; the indication value of any second accumulator being used to represent whether an exception occurs when the execution node executes the task corresponding to the second accumulator based on a first broadcast variable in the execution node; the control node determining an indication value of a first accumulator in the control node based on the indication values of the second accumulators; and the control node updating the first broadcast variable of the control node to a second broadcast variable when determining that the indication value of the first accumulator is abnormal. The performance loss caused by the mode of updating the broadcast variable at a fixed time and the problem of low timeliness of updating are avoided. The computing resource is saved, and the updating is very efficient and timely.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of distributed computing technology, and in particular to a processing method, apparatus, computing device and computer-readable storage medium applied to a distributed computing engine. Background Technology

[0002] In distributed computing engines, streaming data processing often encounters situations where the current broadcast variables are no longer suitable for processing the current task, necessitating dynamic updates to these variables. However, some distributed computing engines do not support dynamic updates of their broadcast variables. For example, Spark's native broadcast variables do not support dynamic updates across execution nodes. Therefore, a timer must be set on the control node to force a fixed-frequency refresh of the broadcast variables, such as deleting the current broadcast variables from the execution nodes every 10 minutes and then reloading the latest broadcast variables from an external source. This allows each execution node to execute tasks based on the updated broadcast variables.

[0003] Refreshing broadcast variables on a timed basis has a certain lag, making it difficult to accurately track changes in actual data content. Moreover, updating broadcast variable values ​​is typically an extremely infrequent operation, and continuously updating broadcast variables in control nodes using a timed method would undoubtedly consume computing power and reduce operational performance.

[0004] To address the above issues, this invention provides a processing method for a distributed computing engine, which efficiently updates broadcast variables dynamically while saving computing resources. Summary of the Invention

[0005] This invention provides a processing method for a distributed computing engine, which can efficiently perform dynamic updates of broadcast variables and save computing resources.

[0006] In a first aspect, embodiments of the present invention provide a processing method applied to a distributed computing engine, comprising:

[0007] The control node obtains the indication value of the second accumulator corresponding to each task in each execution node; the indication value of any second accumulator is used to characterize whether an anomaly occurs when the execution node executes the task corresponding to the second accumulator based on the first broadcast variable in the execution node.

[0008] The control node determines the indication value of the first accumulator in the control node based on the indication values ​​of each second accumulator;

[0009] When the control node determines that the indication value of the first accumulator is abnormal, it updates the first broadcast variable of the control node to a second broadcast variable; the second broadcast variable is used to provide each execution node with a reference when executing tasks in the future.

[0010] By setting a first accumulator in the control node, the first accumulator can obtain the indication values ​​of the second accumulators for each task. When an execution exception occurs in the task corresponding to a certain second accumulator, the indication value of the first accumulator will also be updated. Based on the updated indication value, the first broadcast variable of the task with the execution exception will be updated to the second broadcast variable. Therefore, when the execution nodes execute subsequent tasks based on the second broadcast variable, no errors will occur. This avoids the performance loss and low update timeliness problems caused by periodically updating broadcast variables. It saves computing resources and updates very efficiently and promptly.

[0011] Optionally, the control node determines the indication value of the first accumulator in the control node based on the indication values ​​of each second accumulator, including:

[0012] When the control node determines that the indication value of any second accumulator is abnormal, it sets the indication value of the first accumulator to abnormal.

[0013] When the control node determines that the indication values ​​of each second accumulator are normal, it sets the indication value of the first accumulator to normal.

[0014] If the indicator value of any second accumulator is abnormal, the indicator value of the first accumulator is set to abnormal. This allows for accurate and timely determination of the abnormal state of the first accumulator and timely updating of the second broadcast variable.

[0015] Optionally, the indication value of the second accumulator is determined by:

[0016] For any task in any execution node, if there is an exception handling situation during the execution of the task based on the first broadcast variable, the indicator value of the second accumulator is set to an exception when performing a preset operation; the initial value of the second accumulator is normal.

[0017] Because the distributed computing engine Spark uses a lazy computation mechanism, the corresponding program code is only executed when actual operations (such as counting, merging, and writing) are performed. Therefore, during task execution, if an exception occurs, the indicator value of the second accumulator is determined when performing pre-defined operations. This allows for the acquisition of changes in the indicator value and the updating of broadcast variables based on these changes.

[0018] Optionally, if an exception occurs during the execution of the task based on the first broadcast variable, the execution node sets the indicator value of the second accumulator to an exception when performing a preset operation, including:

[0019] During the execution of the task based on the first broadcast variable, the execution node stores the execution result of the first task in the cache;

[0020] A forced trigger operation is performed on the execution result of the first task in the cache. If it is determined in the forced trigger operation that the execution result of the first task contains an abnormal result, the indicator value of the second accumulator is set to abnormal.

[0021] The execution result of the first task is cached. There may be abnormal results in the execution result of the first task. If there are abnormal results, they will be reflected by the value of the second accumulator in the forced trigger operation. Then the indication value of the first accumulator can be updated, thereby updating the broadcast variable.

[0022] Optionally, after determining the indication value of the first accumulator in the control node, the method further includes:

[0023] When the control node determines that the indication value of the first accumulator is normal, it writes the execution result of the first task in the cache into the database based on the execution node.

[0024] If the value of the first accumulator is normal, then since the result of the first task is pre-stored in the cache, when performing a write operation, there is no need to repeatedly read and process the data. Instead, the result of the first task in the cache can be written directly, saving computing resources and improving write efficiency.

[0025] Optionally, after updating the first broadcast variable of the control node to the second broadcast variable, the method further includes:

[0026] The execution node deletes the abnormal result from the first task execution result in the cache, executes the subtask corresponding to the abnormal result based on the second broadcast variable, and writes the task execution result of the subtask into the cache to obtain the second task execution result;

[0027] The execution node performs a forced trigger operation on the execution result of the second task in the cache. If it is determined in the forced trigger operation that the execution result of the second task does not contain any abnormal results, then the indication value of the second accumulator is determined to be normal.

[0028] When the control node determines that the indication value of the first accumulator in the control node is normal based on the indication value of each second accumulator, the execution node writes the execution result of the second task in the cache into the database.

[0029] When the indicator value of the first accumulator is determined to be abnormal, the first broadcast variable is updated to the second broadcast variable. Since the cache previously stored the execution result of the first task, and the execution result of the first task contained an abnormal result, the subtask corresponding to the abnormal result can be reprocessed based on the second broadcast variable to obtain the execution result of the second task. This corrects the abnormal result in the execution result of the first task, resulting in a more accurate data processing result. Furthermore, in subsequent forced triggering operations, the second accumulator will be determined to be normal because the execution result of the second task does not contain an abnormal result. Therefore, there is no need to update the broadcast variable; instead, the execution result of the second task is written to the database.

[0030] Optionally, when the control node determines that the indication value of the first accumulator is abnormal, it updates the first broadcast variable of the control node to a second broadcast variable, including:

[0031] When the control node determines that the indication value of the first accumulator is abnormal, it obtains the version information of the first broadcast variable.

[0032] The version information of the next broadcast variable is determined based on the version information of the first broadcast variable;

[0033] Based on the version information of the next broadcast variable, the next broadcast variable is loaded from the database, and a second broadcast variable is obtained based on the next broadcast variable and the first broadcast variable; the broadcast variables in the database are updated in real time according to the streaming data in the distributed computing engine.

[0034] In this way, it can be guaranteed that the updated second broadcast variable is correct, and subsequent tasks will be performed based on the correct broadcast variable for data processing.

[0035] Secondly, embodiments of the present invention also provide a processing apparatus for a distributed computing engine, comprising:

[0036] The acquisition unit is used to acquire the indication value of the second accumulator corresponding to each task in each execution node; the indication value of any second accumulator is used to characterize whether an anomaly occurs when the execution node executes the task corresponding to the second accumulator based on the first broadcast variable in the execution node.

[0037] A determining unit is configured to determine the indication value of the first accumulator in the control node based on the indication values ​​of each second accumulator;

[0038] The processing unit is configured to update the first broadcast variable of the control node to a second broadcast variable when it is determined that the indication value of the first accumulator is abnormal; the second broadcast variable is used to provide each execution node with a reference when executing tasks in the future.

[0039] Optionally, the determining unit is specifically used for:

[0040] When the control node determines that the indication value of any second accumulator is abnormal, it sets the indication value of the first accumulator to abnormal.

[0041] When the control node determines that the indication values ​​of each second accumulator are normal, it sets the indication value of the first accumulator to normal.

[0042] Optionally, the acquisition unit is specifically used for:

[0043] For any task in any execution node, if there is an exception handling situation during the execution of the task based on the first broadcast variable, the indicator value of the second accumulator is set to an exception when performing a preset operation; the initial value of the second accumulator is normal.

[0044] Optionally, the acquisition unit is specifically used for:

[0045] During the execution of the task based on the first broadcast variable, the execution node stores the execution result of the first task in the cache;

[0046] A forced trigger operation is performed on the execution result of the first task in the cache. If it is determined in the forced trigger operation that the execution result of the first task contains an abnormal result, the indicator value of the second accumulator is set to abnormal.

[0047] Optionally, the processing unit is further configured to:

[0048] When the control node determines that the indication value of the first accumulator is normal, it writes the execution result of the first task in the cache into the database based on the execution node.

[0049] Optionally, the processing unit is further configured to:

[0050] The execution node deletes the abnormal result from the first task execution result in the cache, executes the subtask corresponding to the abnormal result based on the second broadcast variable, and writes the task execution result of the subtask into the cache to obtain the second task execution result;

[0051] The execution node performs a forced trigger operation on the execution result of the second task in the cache. If it is determined in the forced trigger operation that the execution result of the second task does not contain any abnormal results, then the indication value of the second accumulator is determined to be normal.

[0052] When the control node determines that the indication value of the first accumulator in the control node is normal based on the indication value of each second accumulator, the execution node writes the execution result of the second task in the cache into the database.

[0053] Optionally, the processing unit is specifically used for:

[0054] When the control node determines that the indication value of the first accumulator is abnormal, it obtains the version information of the first broadcast variable.

[0055] The version information of the next broadcast variable is determined based on the version information of the first broadcast variable;

[0056] Based on the version information of the next broadcast variable, the next broadcast variable is loaded from the database, and a second broadcast variable is obtained based on the next broadcast variable and the first broadcast variable; the broadcast variables in the database are updated in real time according to the streaming data in the distributed computing engine.

[0057] Thirdly, embodiments of the present invention also provide a computing device, comprising:

[0058] Memory, used to store computer programs;

[0059] The processor is configured to invoke a computer program stored in the memory and execute the processing methods listed above for application to the distributed computing engine according to the obtained program.

[0060] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer-executable program, the computer-executable program being used to cause a computer to perform the processing methods applied to a distributed computing engine listed in any of the above embodiments. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 A schematic diagram of a Yarn resource scheduling framework provided in an embodiment of the present invention;

[0063] Figure 2 A flowchart illustrating a possible method for dynamically updating broadcast variables according to an embodiment of the present invention;

[0064] Figure 3This is a schematic diagram illustrating a specific method for updating broadcast variables, as provided in an embodiment of the present invention.

[0065] Figure 4 This is a schematic diagram illustrating a possible processing method applied to a distributed computing engine, as provided in an embodiment of the present invention.

[0066] Figure 5 This is a schematic diagram illustrating a possible processing method applied to a distributed computing engine, as provided in an embodiment of the present invention.

[0067] Figure 6 A schematic diagram illustrating the main implementation method of a first accumulator provided in an embodiment of the present invention;

[0068] Figure 7 This is a schematic diagram illustrating a possible processing method applied to a distributed computing engine, as provided in an embodiment of the present invention.

[0069] Figure 8 A schematic diagram of a processing device applied to a distributed computing engine, provided as an embodiment of the present invention;

[0070] Figure 9 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0071] To make the objectives, implementation methods and advantages of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the described exemplary embodiments are only some embodiments of this application, and not all embodiments.

[0072] Based on the exemplary embodiments described in this application, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of the appended claims. Furthermore, although the disclosures in this application are presented by way of one or more exemplary examples, it should be understood that each aspect of these disclosures can also constitute a complete implementation on its own.

[0073] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0074] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities and do not necessarily imply a specific order or sequence, unless otherwise indicated. It should be understood that such terms can be used interchangeably where appropriate, for example, to implement the application in a sequence other than those given in the embodiments illustrated or described herein.

[0075] Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclusively include, for example, a product or device that includes a series of components is not necessarily limited to those that are explicitly listed, but may include other components that are not explicitly listed or that are inherent to such product or device.

[0076] With the development of computing technology, some applications require enormous computing power to complete, which would take a considerable amount of time if centralized computing were used. Distributed computing breaks down the application into many smaller parts and distributes them to multiple computers for processing. This saves overall computing time and greatly improves computing efficiency. Existing distributed computing engines include MapReduce, Flink, and Spark, etc. This invention will use Spark as an example for illustration.

[0077] Spark, as a distributed computing framework, typically runs its application processes across a series of independent computing nodes. To manage and allocate computing resources uniformly, these nodes are usually grouped into a cluster, and a cluster administrator allocates the computing resources required by the application. Common cluster resource scheduling frameworks include Mesos, YARN, and Kubernetes.

[0078] Figure 1 An exemplary embodiment of the Yarn resource scheduling framework applicable to this invention is illustrated. This framework includes a client, a resource manager, a control node, and several execution nodes. The following is a brief description of how resource allocation and scheduling are performed.

[0079] 1. Users submit jobs through the client to request the resource manager of the Yarn cluster.

[0080] 2. After receiving a job request, the resource manager randomly selects a host and sends the commands and job information (as the control node) to that host. For example, if host 1 is selected, host 1, upon receiving the commands, will create a container to act as the control node. The control node is only responsible for job allocation and scheduling and does not perform the actual data processing tasks.

[0081] 3. The control node starts the application management service process and then performs Spark Context initialization. After initialization, the DAG scheduler constructs a DAG (Directed Acyclic Graph) from the jobs received in step 2.

[0082] 4. After the DAG graph is successfully created, the Yarn cluster scheduler requests several containers from the resource manager to serve as execution nodes based on the DAG graph.

[0083] 5. After receiving the request, the resource manager randomly selects several hosts and assigns containers as execution nodes. For example, if hosts 2 and 3 are selected, two containers (two execution nodes) are set up on host 2, and one container (one execution node) is set up on host 3.

[0084] 6. The selected host starts each execution node, which is used to perform the actual task according to the instructions of the control node.

[0085] 7. In application management, the Yarn cluster scheduler allocates tasks to the created execution nodes based on the constructed DAG graph and receives the execution results from the execution nodes. Tasks on each execution node are executed in parallel.

[0086] like Figure 1 As shown, a Spark application splits a job into several independent tasks, which then run in parallel across multiple independent execution nodes. Each execution node should process the tasks using a consistent processing method. To unify the processing method across each execution node, we provide a method using broadcast variables.

[0087] Data that needs to be shared across multiple execution nodes is defined as a broadcast variable. This broadcast variable is generated at the control node, and each execution node can retrieve it from the control node and cache a copy of the broadcast variable's data. Data processing is then performed based on this copy. To ensure consistent execution across all tasks, the value of the broadcast variable cannot be modified at the execution node; modification is only permitted at the control node.

[0088] In practical applications of streaming data processing, there is often a need to dynamically update broadcast variables, but the broadcast variables provided by Spark natively do not support dynamic updates at the execution node. Figure 2This paper illustrates a possible method for dynamically updating broadcast variables. A timer is set in the control node to force a fixed-frequency refresh of the broadcast variables, such as deleting the current data copy of the broadcast variables from the execution nodes every 10 minutes. Then, the control node reloads the latest broadcast variables from an external source. Subsequently, when an execution node determines that a task cannot be executed, it retrieves the latest broadcast variables from the control node and executes the task according to the updated broadcast variables. To save network bandwidth, each execution node, upon determining that a task cannot be executed, can first check if the broadcast variables exist in other execution nodes on the same host. If so, it retrieves them from those other execution nodes on the same host, without needing to retrieve them from the control node.

[0089] For example, Figure 3 An example of updating a broadcast variable is shown. Execution node 1 is used to convert a series of streaming data into a corresponding data format, and the broadcast variable is used to indicate the data format.

[0090] The external database stores data formats that are updated in real time as the streaming data flows. For example, when the streaming data is A1, A2, A3, A1, B1, B2, B3, C1, the data format in the database is s1; when the streaming data is A4, C2, C3, a new data format s2 is added to the database; when the streaming data is C4, a new data format s3 is added to the database, and so on.

[0091] exist Figure 3 In this scenario, execution node 1 has 3 CPUs and can process 3 tasks. The current broadcast variable is s1: name: String, age: Integer. Therefore, execution node 1 stores a data copy of the broadcast variable as s1. When executing a task based on this data copy, if the obtained stream data contains the name (name), it is converted to a string (String); if the obtained stream data contains the age (age), it is converted to an integer (Integer). For example, task 1 obtains stream data A1 and A2, converts the name to a string and the age to an integer; task 2 obtains stream data B1, converts the name to a string and the age to an integer; task 3 obtains stream data C1, converts the name to a string and the age to an integer.

[0092] When Task 3 processes stream data C2, the acquired stream data contains not only name and age, but also gender. Since s1 did not define the data format for gender, the stream data C2 failed to be converted.

[0093] If broadcast variables are forced to be refreshed at a fixed frequency, such as once every 10 minutes, then after 10 minutes, the data copy s1 of the current broadcast variable in execution node 1 is deleted first, and then the control node reloads the latest broadcast variable s2 from the external database. When execution node 1 confirms that the task cannot be executed, it obtains the broadcast variable s2 from the control node, so that subsequent tasks can be executed based on s2.

[0094] The above-mentioned broadcast variable update mechanism has the following problems:

[0095] 1. Broadcast variables have low timeliness in updating.

[0096] Refreshing broadcast variables using a timed method has a certain lag, making it difficult to accurately and dynamically track changes in actual data content.

[0097] 2. Excessive resource consumption.

[0098] Generally speaking, updating broadcast variables is an extremely infrequent operation requirement. The program does not need to update broadcast variables for most of the time. Furthermore, since data broadcasting is an extremely inefficient and time-consuming operation, using a timed method to refresh broadcast variables in the control node will reduce the performance of the application to some extent.

[0099] To address the aforementioned problems, embodiments of the present invention provide a possible processing method applied to a distributed computing engine, such as... Figure 4 As shown, it includes:

[0100] Step 401: The control node obtains the indication value of the second accumulator corresponding to each task in each execution node; the indication value of any second accumulator is used to characterize whether an anomaly occurs when the execution node executes the task corresponding to the second accumulator based on the first broadcast variable in the execution node.

[0101] An accumulator is set up in each task of each execution node; for ease of distinction from the following text, it will be named the second accumulator here. For example, in Figure 3 In the example, Task 1, Task 2 and Task 3 each have a second accumulator.

[0102] An accumulator is a variable used only for addition operations. It provides two function interfaces, `add` and `merge`. Within a task, the value of the accumulator can be modified using the `add` operation, and the values ​​of accumulators from all tasks can be merged using the `merge` operation. Accumulators can be used to perform operations such as counting and summing, for example, to count the total number of records and the total processing time across all distributed execution nodes.

[0103] Accumulators can also be used to generate indicator values ​​based on the task's execution status. For example, in... Figure 3 In the example, execution node 1 can execute tasks 1, 2, and 3 in parallel. Each of tasks 1, 2, and 3 has a second accumulator. The current first broadcast variable is s1. When executing A1 and A2 of task 1, no execution exception occurs, so the second accumulator indicator value of task 1 is "normal". When executing B1 of task 2, no execution exception occurs, so the second accumulator indicator value of task 2 is "normal". When executing C1 of task 3, no execution exception occurs, so the second accumulator indicator value of task 3 is "normal". When executing C2 of task 3, because the data format of gender cannot be obtained from the first broadcast variable, it is impossible to determine which format to convert gender to, thus an execution exception occurs, and the second accumulator indicator value of task 3 is updated to "exception".

[0104] A first accumulator is set in the control node. By calling the merge function of the first accumulator, the indicator values ​​of each second accumulator can be obtained. For example, the indicator value of the second accumulator for Task 1 is "normal", the indicator value of the second accumulator for Task 2 is "normal", and the indicator value of the second accumulator for Task 3 is "abnormal".

[0105] Optionally, the control node can be configured to call the merge function after a certain stage is completed, or the execution node can be configured to call the merge function after processing a portion of the data. This embodiment of the invention does not impose any limitations on this. Here, a stage refers to reading data, processing data, and writing data. Due to the lazy calculation mechanism of the accumulator, generally, the accumulator will only execute the corresponding program code when an actual trigger operation (count, collect, write, etc.) occurs. The second accumulator updates its indicator value, and the control node calls the merge function to obtain the indicator values ​​of each second accumulator.

[0106] Step 402: The control node determines the indication value of the first accumulator in the control node based on the indication values ​​of each second accumulator.

[0107] Specifically, when the control node determines that the indication value of any second accumulator is abnormal, it sets the indication value of the first accumulator to abnormal; when the control node determines that the indication values ​​of all second accumulators are normal, it sets the indication value of the first accumulator to normal.

[0108] For example, in the above example, since the second accumulator indicator value of Task 3 is "abnormal", the indicator value of the first accumulator in the control node is also abnormal.

[0109] Step 403: When the control node determines that the indication value of the first accumulator is abnormal, it updates the first broadcast variable of the control node to a second broadcast variable; the second broadcast variable is used to provide each execution node with a reference when executing tasks in the future.

[0110] After determining that the indication value of the first accumulator is abnormal, it indicates that the task of an execution node cannot perform normal data processing based on the current first broadcast variable. Therefore, the first broadcast variable is updated to the second broadcast variable, so that subsequent execution nodes can execute tasks based on the second broadcast variable.

[0111] For example, in the example above, when the control node determines that the indication value of the first accumulator is abnormal, it loads the data format s2 that is updated in real time with the streaming data from the database and determines s2 as the second broadcast variable.

[0112] Alternatively, s2 and s1 can be merged as the second broadcast variable. Since s2 and s1 may not be exactly the same, merging them will result in a more comprehensive second broadcast variable, which is beneficial for reducing the error rate in data processing.

[0113] Optionally, since the data format in the database is updated in real time along with the streaming data, the control node may not obtain the indication values ​​of the second accumulators corresponding to each task in each execution node in real time. For example, the execution node may obtain the indication values ​​of each second accumulator after completing a step (reading data, processing data, writing data), thereby determining the indication value of the first accumulator and updating the broadcast variable. Therefore, when the control node determines that the indication value of the first accumulator is abnormal, the data format in the database may have already been updated to s5, while we actually need to obtain s2. Therefore, when the control node determines that the indication value of the first accumulator is abnormal, it can also obtain the version information of the first broadcast variable; determine the version information of the next broadcast variable based on the version information of the first broadcast variable; load the next broadcast variable from the database based on the version information of the next broadcast variable; and obtain the second broadcast variable based on the next broadcast variable and the first broadcast variable.

[0114] For example, if the version information of the first broadcast variable is obtained as s1, the version information of the next broadcast variable can be determined as s2. Then s2 can be directly determined as the second broadcast variable, or s1 and s2 can be merged to obtain the second broadcast variable.

[0115] The method provided by the embodiments of the present invention will be further described below through specific examples.

[0116] Example 1

[0117] Figure 5This illustrates a possible processing method for distributed computing engines, where message producers add streaming data to be processed to a message queue. Figure 5 In the example, the message queue has 3 partitions, and execution node 1 has 3 CPUs, which can process 3 tasks simultaneously. Because Spark uses a lazy computation mechanism, the execution node only retrieves streaming data from the message queue for processing when it performs the actual trigger operation, i.e., writing data. Taking task 1 as an example, the execution node first reads A1 and A2, then processes the data, converting the streaming data into the appropriate data format, and then writes the data to the storage group. During this process, no exceptions occurred during the execution of task 1 based on the first broadcast variable; therefore, the indicator value of the second accumulator for task 1 is normal.

[0118] Taking Task 3 as an example, the execution node first reads C1 and C2, then processes the data, converting the streaming data into the corresponding data format and writing the data to the storage group. During this process, when executing the C2 data of Task 3 based on the first broadcast variable, it is determined that there is no corresponding converted data format for the gender field, resulting in an exception. Therefore, the indicator value of the second accumulator in Task 3 is abnormal. The execution node can write the processing result to the storage group. The processing result can include only the successfully processed C1, or it can include both the successfully processed C1 and the unprocessed C2.

[0119] It is worth mentioning that, due to Spark's lazy computation mechanism, the execution node will only update the indicator value of the second accumulator when it performs a preset operation. The preset operation here is the actual triggering operation mentioned above. In this specific embodiment, the preset operation is the operation of writing data.

[0120] In this example, one step involves reading data, processing data, and writing data. After one step is completed, the control node calls the `merge` function to obtain the indicator values ​​of each second accumulator. The indicator values ​​of the second accumulator for Task 1 and Task 2 are normal, while those for Task 3 are abnormal. Therefore, the indicator value of the first accumulator is abnormal, and it is updated to abnormal. The node then checks if the indicator value has changed; in this example, it has changed. Therefore, it proceeds to step 4, updating the broadcast variable and obtaining the current first broadcast variable as `s1`. Based on `s1`, the next broadcast variable is determined to be `s2`. Therefore, `s2` is loaded from the external database and added to the control node's broadcast variable as the second broadcast variable. Thus, subsequent data readings, i.e., in step 5, will be based on the second broadcast variable for data reading and processing.

[0121] exist Figure 5In this context, the version information of the first accumulator is determined based on the version information of the broadcast variable. For example, if the version information of the broadcast variable is s1, the version information of the first accumulator is 1.

[0122] The following is a brief introduction to the main implementation of the first accumulator. Figure 6 As shown, attribute 1 is used to identify whether the broadcast variable needs to be updated; for example, if the indicator value of the first accumulator is determined to be abnormal, then attribute 1 is set to yes. Attribute 2 is used to record the version information of the current broadcast variable, for example, in... Figure 5 In the example, the current version of the broadcast variable is s1; the first method 1 is used to get the current version information s1 of the broadcast variable through attribute 2; the second method 1 is used to get whether the broadcast variable needs to be updated through attribute 1; the third method 1 is used to update the version information of the broadcast variable. Figure 5 In the example, the version information of the broadcast variable is updated to s2; Method 2 is used to merge the indication values ​​of the second accumulator of all tasks.

[0123] As can be seen, in the above embodiments, the update of the broadcast variable is indicated by the indicator value of the first accumulator, which eliminates the need for periodic refresh. This reduces the system load and allows for more timely and flexible adjustment of the broadcast variable, thereby ensuring that data processing proceeds normally and in an orderly manner.

[0124] However, adjusting broadcast variables using the above method is not timely enough. Because Spark uses a lazy calculation mechanism, the second accumulator's indicator value is only updated when an actual trigger operation occurs. It cannot immediately update the second accumulator after a processing error occurs, resulting in a certain lag. Furthermore, data not processed based on the correct second broadcast variable is not reprocessed and therefore not corrected. For example, in the case of C2 above, because C2 processed data based on the first broadcast variable, it detected a processing error and updated the second broadcast variable, but C2 did not recalculate based on the second broadcast variable again; therefore, the data in C2 was not corrected.

[0125] Example 2

[0126] Figure 7 This illustrates a possible processing method for distributed computing engines, where message producers add streaming data to be processed to a message queue. Figure 7 In the example, the message queue has 3 partitions, and execution node 1 has 3 CPUs, which can process 3 tasks at the same time.

[0127] To address the problem in Example 1, we add a forced trigger operation before writing data. This forced trigger operation can be a counting, settling, or other actual trigger operation. Figure 7In Spark, due to its lazy computation mechanism, execution nodes only retrieve streaming data from the message queue for processing when they perform the actual triggering operation (counting operation). After processing, the execution result of the first task is stored in the cache. When the counting operation is performed on the first task's execution result in the cache, the second accumulator is updated. The indicator value of the second accumulator for task 1 is updated to normal, the indicator value of the second accumulator for task 2 is updated to normal, and the indicator value of the second accumulator for task 3 is updated to abnormal. In this example, one step refers to reading data - processing data - caching processing - counting operation. After one step is completed, the control node calls the merge function to merge the indicator values ​​of each second accumulator, resulting in the first accumulator's indicator value being abnormal. Since the first accumulator's indicator value has changed, the first broadcast variable s1 is updated to the second broadcast variable s2, i.e., step 5.1. Simultaneously with updating the broadcast variable, step 5.2 is executed to delete the abnormal result from the first task's execution result in the cache. Based on the modified second broadcast variable, the subtask corresponding to the abnormal result is executed, and the subtask's execution result is written into the cache to obtain the second task's execution result. For example, in this case, the first execution result of Task 3 includes successfully processed C1 and unsuccessfully processed C2. Therefore, C2 is the abnormal result. Based on the second broadcast variable, the task execution result of C2 is written into the cache, resulting in the second task execution result. The second task execution result then includes both successfully processed C1 and successfully processed C2. When subsequent execution nodes perform a counting operation on the second task execution result, they can find that the cached second task execution result does not contain abnormal results. Therefore, the indicator value of the second accumulator is normal, and the indicator value of the first accumulator is also normal. Since the indicator value remains unchanged, step 6 can be executed to write the data in the cache to the storage repository and clear the cache.

[0128] In this specific embodiment, the reason for setting step 3 before step 4 to cache the processed first task execution result is that when the indication value of the first accumulator is normal, the data in the cache can be directly written when the write operation of step 6 is executed, without having to perform data reading and data processing operations again (because writing data is the actual triggering operation, and data will be read again when writing data is executed), thus saving a lot of computing resources.

[0129] Based on the same technological concept Figure 8 An exemplary embodiment of the present invention illustrates the structure of a processing apparatus applied to a distributed computing engine, which can execute the processing flow applied to the distributed computing engine.

[0130] like Figure 8 As shown, the device specifically includes:

[0131] The acquisition unit 801 is used to acquire the indication value of the second accumulator corresponding to each task in each execution node; the indication value of any second accumulator is used to characterize whether an abnormality occurs when the execution node executes the task corresponding to the second accumulator based on the first broadcast variable in the execution node.

[0132] The determining unit 802 is used to determine the indication value of the first accumulator in the control node based on the indication values ​​of each second accumulator;

[0133] The processing unit 803 is configured to update the first broadcast variable of the control node to a second broadcast variable when it is determined that the indication value of the first accumulator is abnormal; the second broadcast variable is used to provide each execution node with a reference when executing a task in the future.

[0134] Optionally, the determining unit 802 is specifically used for:

[0135] When the control node determines that the indication value of any second accumulator is abnormal, it sets the indication value of the first accumulator to abnormal.

[0136] When the control node determines that the indication values ​​of each second accumulator are normal, it sets the indication value of the first accumulator to normal.

[0137] Optionally, the acquisition unit 801 is specifically used for:

[0138] For any task in any execution node, if there is an exception handling situation during the execution of the task based on the first broadcast variable, the indicator value of the second accumulator is set to an exception when performing a preset operation; the initial value of the second accumulator is normal.

[0139] Optionally, the acquisition unit 801 is specifically used for:

[0140] During the execution of the task based on the first broadcast variable, the execution node stores the execution result of the first task in the cache;

[0141] A forced trigger operation is performed on the execution result of the first task in the cache. If it is determined in the forced trigger operation that the execution result of the first task contains an abnormal result, the indicator value of the second accumulator is set to abnormal.

[0142] Optionally, the processing unit 803 is further configured to:

[0143] When the control node determines that the indication value of the first accumulator is normal, it writes the execution result of the first task in the cache into the database based on the execution node.

[0144] Optionally, the processing unit 803 is further configured to:

[0145] The execution node deletes the abnormal result from the first task execution result in the cache, executes the subtask corresponding to the abnormal result based on the second broadcast variable, and writes the task execution result of the subtask into the cache to obtain the second task execution result;

[0146] The execution node performs a forced trigger operation on the execution result of the second task in the cache. If it is determined in the forced trigger operation that the execution result of the second task does not contain any abnormal results, then the indication value of the second accumulator is determined to be normal.

[0147] When the control node determines that the indication value of the first accumulator in the control node is normal based on the indication value of each second accumulator, the execution node writes the execution result of the second task in the cache into the database.

[0148] Optionally, the processing unit 803 is specifically used for:

[0149] When the control node determines that the indication value of the first accumulator is abnormal, it obtains the version information of the first broadcast variable.

[0150] The version information of the next broadcast variable is determined based on the version information of the first broadcast variable;

[0151] Based on the version information of the next broadcast variable, the next broadcast variable is loaded from the database, and a second broadcast variable is obtained based on the next broadcast variable and the first broadcast variable; the broadcast variables in the database are updated in real time according to the streaming data in the distributed computing engine.

[0152] Based on the same technical concept, embodiments of this application provide a computer device, such as... Figure 9 As shown, it includes at least one processor 901 and a memory 902 connected to at least one processor. In this embodiment, the specific connection medium between the processor 901 and the memory 902 is not limited. Figure 9 Taking the connection between processor 901 and memory 902 via a bus as an example, the bus can be divided into address bus, data bus, control bus, etc.

[0153] In this embodiment of the application, the memory 902 stores instructions that can be executed by at least one processor 901. By executing the instructions stored in the memory 902, at least one processor 901 can perform the steps of the processing method applied to the distributed computing engine described above.

[0154] The processor 901 is the control center of the computer device, capable of connecting to various parts of the computer device via various interfaces and lines. It performs processing for the distributed computing engine by running or executing instructions stored in the memory 902 and accessing data stored in the memory 902. Optionally, the processor 901 may include one or more processing units. The processor 901 may integrate an application processor and a modem processor. The application processor primarily handles the operating system, user interface, and applications, while the modem processor primarily handles wireless communication. It is understood that the modem processor may not be integrated into the processor 901. In some embodiments, the processor 901 and the memory 902 may be implemented on the same chip; in other embodiments, they may be implemented on separate chips.

[0155] The processor 901 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0156] Memory 902, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 902 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. Memory 902 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. In the embodiments of this application, memory 902 can also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0157] Based on the same technical concept, embodiments of the present invention also provide a computer-readable storage medium storing a computer-executable program, the computer-executable program being used to cause a computer to perform the processing methods listed above for application to a distributed computing engine.

[0158] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0159] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0160] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0161] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0162] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A processing method applied to a distributed computing engine, characterized in that, include: The control node obtains the indication value of the second accumulator corresponding to each task in each execution node; The indication value of any second accumulator is used to characterize whether an anomaly occurs when the execution node executes the task corresponding to the second accumulator based on the first broadcast variable in the execution node; When the control node determines that the indication value of any of the second accumulators is abnormal, it sets the indication value of the first accumulator to abnormal. When the control node determines that the indication values ​​of each second accumulator are normal, it sets the indication value of the first accumulator to normal. When the control node determines that the indication value of the first accumulator is abnormal, it updates the first broadcast variable of the control node to a second broadcast variable; the second broadcast variable is used to provide each execution node with a reference when executing tasks in the future.

2. The method as described in claim 1, characterized in that, The indication value of the second accumulator is determined in the following ways: For any task in any execution node, if there is an exception handling situation during the execution of the task based on the first broadcast variable, the indicator value of the second accumulator is set to an exception when performing a preset operation; the initial value of the second accumulator is normal.

3. The method as described in claim 2, characterized in that, If an exception occurs during the execution of the task based on the first broadcast variable by the execution node, the indicator value of the second accumulator will be set to an exception when performing a preset operation, including: During the execution of the task based on the first broadcast variable, the execution node stores the execution result of the first task in the cache; A forced trigger operation is performed on the execution result of the first task in the cache. If it is determined in the forced trigger operation that the execution result of the first task contains an abnormal result, the indicator value of the second accumulator is set to abnormal.

4. The method as described in claim 3, characterized in that, After determining the indication value of the first accumulator in the control node, the method further includes: When the control node determines that the indication value of the first accumulator is normal, it writes the execution result of the first task in the cache into the database based on the execution node.

5. The method as described in claim 3, characterized in that, After updating the first broadcast variable of the control node to the second broadcast variable, the method further includes: The execution node deletes the abnormal result from the first task execution result in the cache, executes the subtask corresponding to the abnormal result based on the second broadcast variable, and writes the task execution result of the subtask into the cache to obtain the second task execution result; The execution node performs a forced trigger operation on the execution result of the second task in the cache. If it is determined in the forced trigger operation that the execution result of the second task does not contain any abnormal results, then the indication value of the second accumulator is determined to be normal. When the control node determines that the indication value of the first accumulator in the control node is normal based on the indication value of each second accumulator, the execution node writes the execution result of the second task in the cache into the database.

6. The method as described in claim 1, characterized in that, When the control node determines that the indication value of the first accumulator is abnormal, it updates the first broadcast variable of the control node to a second broadcast variable, including: When the control node determines that the indication value of the first accumulator is abnormal, it obtains the version information of the first broadcast variable. The version information of the next broadcast variable is determined based on the version information of the first broadcast variable; Based on the version information of the next broadcast variable, the next broadcast variable is loaded from the database, and a second broadcast variable is obtained based on the next broadcast variable and the first broadcast variable; the broadcast variables in the database are updated in real time according to the streaming data in the distributed computing engine.

7. A processing device applied to a distributed computing engine, characterized in that, include: The acquisition unit is used to acquire the indication value of the second accumulator corresponding to each task in each execution node; The indication value of any second accumulator is used to characterize whether an anomaly occurs when the execution node executes the task corresponding to the second accumulator based on the first broadcast variable in the execution node; A determining unit is configured to set the indicator value of the first accumulator to abnormal when it is determined that the indicator value of any second accumulator is abnormal; Once it is determined that the indication values ​​of each of the second accumulators are normal, the indication value of the first accumulator is set to normal. The processing unit is configured to update the first broadcast variable of the control node to a second broadcast variable when it is determined that the indication value of the first accumulator is abnormal; the second broadcast variable is used to provide each execution node with the option to retrieve it during subsequent task execution.

8. A computing device, characterized in that, include: Memory, used to store computer programs; A processor is configured to invoke a computer program stored in the memory and execute the method according to any one of claims 1 to 6 in accordance with the obtained program.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer-executable program for causing a computer to perform the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product stores instructions that, when read and executed by a computer, cause the computer to perform the method as described in any one of claims 1 to 6.

Images