Data processing method, execution workstation, electronic device and storage medium

By combining task results in multiple processing cores in the execution workstation and reducing when conditions are met, the problem of large storage, processing and communication overhead under the Spark execution model is solved, and the performance of big data processing is improved.

CN115599507BActive Publication Date: 2024-11-05TSINGHUA UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110767689.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-07-07
Publication Date
2024-11-05
Estimated Expiration
2041-07-07

AI Technical Summary

Technical Problem

The existing big data processing framework has the problem of excessive storage, processing and communication overhead in distributed computing. Especially under the Spark execution model, each CPU core of the executor needs to serialize the results and store them after completing the task, resulting in wasted resources. and poor performance.

Method used

By introducing multiple processing cores in the execution workstation, receiving tasks and generating task results with predetermined data structures, combining them into shared internal memory, and using shared task results with other execution workstations when conditions are met , reduce storage, processing and communication overhead.

Benefits of technology

It reduces data storage, processing and communication overhead, improves data processing performance, reduces resource waste, and achieves better system performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115599507B_ABST
    Figure CN115599507B_ABST
Patent Text Reader

Abstract

A data processing method, an execution workstation, an electronic device, and a computer-readable storage medium. The data processing method comprises: receiving a task assigned to each of a plurality of processing cores from a management workstation; each of the plurality of processing cores respectively executing the assigned task, and generating a task result having a predetermined data structure after each task is executed; merging the task result generated by each processing core each time into a shared task result stored in an internal memory of the execution workstation, the shared task result having the same data structure as the task result generated by each processing core each time; and when a predetermined condition is met, using the shared task result to reduce with the task results of other execution workstations. The method merges the task results in the same execution workstation in the execution workstation before reducing with the task results in other execution workstations, thereby reducing storage, processing, and communication overheads.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Embodiments of the present disclosure relate to a data processing method, an execution workstation, an electronic device, and a computer-readable storage medium. Background Art

[0002] Today is the data age, and data informatization is closely related to our lives and work. Due to the rapid increase in the amount of data, it is difficult for a single machine to process it, so a big data processing framework is designed for distributed computing. Existing big data processing frameworks include Hadoop, Storm, Samza, Flink, Spark, etc., among which Spark is one of the most popular big data processing frameworks today. Spark is a big data processing framework based on memory computing, which improves the real-time performance of data processing in a big data environment, while providing good horizontal scalability and fault-tolerant processing mechanisms. Spark introduces the abstraction of resilient distributed datasets (RDDs), which are special collections with fault-tolerant mechanisms. If part of a dataset is lost, they can be rebuilt according to the data derivation process. Since RDDs can be transformed into other RDDs through transformation operations, these transformation operations will be recorded. Therefore, Spark uses a lineage graph to track the dependencies between RDDs and recalculate the lost part of the data through the dependencies instead of recalculating all the data. If the lineage chain is very long or the dependencies are too wide, it may not be possible to rerun in the event of a failure. The solution is to set checkpoints for such RDDs. Using lineage and checkpoint technology, Spark can achieve fault tolerance and recovery. Summary of the invention

[0003] At least one embodiment of the present disclosure provides a data processing method for distributed computing executed by an execution workstation, the execution workstation including multiple processing cores, the data processing method including: receiving a task assigned to each of the multiple processing cores from a management workstation; each of the multiple processing cores respectively executing the assigned task and generating a task result with a predetermined data structure after executing each task; merging the task result generated by each processing core each time the task is executed into a shared task result stored in an internal memory of the execution workstation, the shared task result having the same data structure as the task result generated by each processing core each time the task is executed; and when a predetermined condition is met, using the shared task result to reduce with the task results of other execution workstations.

[0004] For example, in the data processing method provided by at least one embodiment of the present disclosure, the predetermined condition is that the shared task result has merged a predetermined number of task results.

[0005] For example, in the data processing method provided by at least one embodiment of the present disclosure, the predetermined condition is receiving an instruction from a management workstation.

[0006] For example, in the data processing method provided by at least one embodiment of the present disclosure, the method further includes sending the completion status of each task to the management workstation; the instruction is an instruction issued by the management workstation when it determines that all tasks of the current task group have been completed based on the status of each task.

[0007] For example, in the data processing method provided by at least one embodiment of the present disclosure, merging the task results generated by each processing core each time it is executed into a shared task result stored in the internal memory of the execution workstation includes: generating a shared task result having a predetermined data structure and an initial value of 0 in the internal memory; and after any processing core completes executing a task and generates a task result, merging the generated task result with the currently stored shared task result, and updating the shared task result with the merged result.

[0008] For example, in the data processing method provided by at least one embodiment of the present disclosure, merging the task results generated by each processing core each time it executes into the shared task results stored in the internal memory of the execution workstation includes: storing the task results generated for the first time when multiple processing cores process the tasks of the current task group as the initial shared task results; and after any processing core completes the execution of a task and generates a task result, merging the generated task result with the currently stored shared task result, and updating the shared task result with the merged result.

[0009] For example, in the data processing method provided by at least one embodiment of the present disclosure, using the shared task result for reduction with the task results of other execution workstations includes: serializing the shared task result and sending it to other execution workstations or management workstations; or receiving the task result from other execution workstations and reducing it with the shared task result.

[0010] For example, in the data processing method provided by at least one embodiment of the present disclosure, when the predetermined condition is met, the shared task result is also stored in a non-volatile storage device.

[0011] At least one embodiment of the present disclosure provides an execution workstation for distributed computing, the execution workstation includes a receiving module, multiple processing cores, a merging module, an internal memory and a result processing module, wherein: the receiving module is used to receive tasks assigned to each of the multiple processing cores from a management workstation; each of the multiple processing cores is used to execute the assigned tasks respectively, and generate a task result with a predetermined data structure after executing each task; the merging module is used to merge the task results generated by each processing core each time they are executed into a shared task result stored in the internal memory of the execution workstation, the shared task result and the task result generated by each processing core each time they are executed have the same data structure; and the result processing module is used to use the shared task result to reduce with the task results of other execution workstations when predetermined conditions are met.

[0012] At least one embodiment of the present disclosure provides an electronic device, including a processor and a memory, wherein the memory stores one or more computer program instructions, wherein the one or more computer program instructions are stored in the memory and, when executed by the processor, implement instructions of the data processing method provided by any embodiment of the present disclosure.

[0013] At least one embodiment of the present disclosure provides a computer-readable storage medium for storing non-transitory computer-readable instructions. When the non-transitory computer-readable instructions are executed by a computer, the data processing method provided by any embodiment of the present disclosure can be implemented.

[0014] The data processing method, execution workstation, electronic device and computer-readable storage medium according to the embodiments of the present disclosure can reduce data storage, processing and communication overhead and improve data processing performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present disclosure, but are not intended to limit the present disclosure.

[0016] Figure 1 A system architecture applied to a data processing method provided by at least one embodiment of the present disclosure is shown;

[0017] Figure 2 A flowchart of a data processing method provided by at least one embodiment of the present disclosure is shown;

[0018] Figure 3 It shows that at least one embodiment of the present disclosure provides Figure 2 The method flow chart of step S203;

[0019] Figure 4 It shows that at least one embodiment of the present disclosure provides Figure 2 Another method flow chart of step S203;

[0020] Figure 5 A schematic block diagram of an execution workstation for distributed computing provided by at least one embodiment of the present disclosure is shown;

[0021] Figure 6 A schematic block diagram of an electronic device provided by at least one embodiment of the present disclosure is shown;

[0022] Figure 7 A schematic block diagram showing another electronic device provided by at least one embodiment of the present disclosure; and

[0023] Figure 8 A schematic diagram of a computer-readable storage medium provided by at least one embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0025] Unless otherwise defined, the technical terms or scientific terms used in the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, similar words such as "one", "one" or "the" do not indicate quantity restrictions, but indicate that there is at least one. Similar words such as "include" or "comprise" mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Similar words such as "connect" or "connected" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0026] Taking the big data processing framework Spark as an example, Spark allows a single executor to use multiple CPU cores in its resource model. Therefore, multiple tasks will be scheduled to the same executor in the same stage. Under the existing Spark execution model, each CPU core of the executor will serialize its results into byte arrays after completing each task, store them in the hard disk and send them to the drive. In this case, the executor generates large storage, processing and communication overheads, and the inventors have found that the serialization process generates a large processing overhead. Therefore, it is important to reduce storage, processing and communication overheads to obtain better performance.

[0027] At least one embodiment of the present disclosure provides a data processing method, an execution workstation, an electronic device, and a computer-readable storage medium for distributed computing. The data processing method includes: receiving a task assigned to each of a plurality of processing cores from a management workstation; each of the plurality of processing cores respectively performs the assigned task, and generates a task result with a predetermined data structure after each task is executed; merging the task result generated by each processing core each time into a shared task result stored in an internal memory of the execution workstation, the shared task result having the same data structure as the task result generated by each processing core each time; and when a predetermined condition is met, using the shared task result for reduction with the task results of other execution workstations.

[0028] The data processing method of this embodiment can merge the task results in the same execution workstation before reducing them with the task results in other execution workstations, thereby reducing storage, processing and communication overheads and achieving better performance.

[0029] It should be noted that the big data processing framework according to the embodiments of the present disclosure includes but is not limited to Spark, and the data processing method provided by at least one embodiment of the present disclosure can also be applied to other big data processing frameworks.

[0030] Figure 1 A system architecture 100 applied to a data processing method provided by at least one embodiment of the present disclosure is shown.

[0031] like Figure 1As shown, the system architecture 100 may include a management workstation 101 and an execution workstation 102, wherein the execution workstation includes multiple processing cores 103 and an internal memory 104. The execution workstation 102 is a computing device for task processing in a distributed computing system, for example, it can run an executor in a Spark architecture; the management workstation 101 is a computing device responsible for managing each execution workstation 102 in a distributed computing system, for example, it can run a driver in a Spark architecture, and the management workstation 101 can coordinate, schedule, and monitor the tasks of each execution workstation. The task results of the execution workstation 102 can eventually be summarized in the management workstation 101.

[0032] Multiple processing cores 103 may be configured in one execution workstation 102. Multiple processing cores 103 in the same execution workstation 102 share an internal memory 104, which is used to store shared task results. The internal memory 104 is also called a memory, which is a memory for temporarily storing programs and data when a computing device is running. It can be any form of volatile memory, such as a random access memory (RAM), a cache, etc. Each of the multiple processing cores 103 can send a message to the management workstation 101 to request the assignment of a task, and the management workstation 101 assigns the task to the processing core 103 in response to the request. Alternatively, the management workstation 101 can also actively assign tasks to the processing core 103. Each of the multiple processing cores 103 receives the assigned task from the management workstation 101 and executes it. Each of the multiple processing cores 103 generates a task result after executing a task. The task results generated by each of the multiple processing cores 103 after executing a task are merged into the shared task results stored in the internal memory 104 in the execution workstation 102.

[0033] The management workstation 101 can interact with the execution workstation 102 through the communication network, and the execution workstation 102 can interact with other execution workstations 102 through the communication network to receive or send messages. The communication network is used to provide a medium for communication links between the management workstation 101 and multiple processing cores 103 and between multiple execution workstations 102. The communication network may include various connection types, such as wired or wireless communication links, specifically WIFI, 3G, 4G, 5G, and fiber optic cables, etc.

[0034] The management workstation 101 is responsible for central coordination, scheduling the processing cores 103 in each execution workstation 102, and monitoring the execution of tasks in each processing core 103. Each of the multiple processing cores 103 executes a task and reports the execution status and progress to the management workstation 101, so that the management workstation 101 can grasp the execution status of each task and restart the task when a task fails.

[0035] Figure 2 A flow chart of a data processing method for distributed computing provided by at least one embodiment of the present disclosure and executable by an execution workstation is shown. The execution workstation includes multiple processing cores.

[0036] like Figure 2 As shown, the data processing method includes steps S201 to S204.

[0037] Step S201: receiving a task assigned to each of a plurality of processing cores from a management workstation.

[0038] Step S202: Each of the multiple processing cores executes the assigned tasks respectively, and generates a task result with a predetermined data structure after executing each task.

[0039] Step S203: merging the task results generated by each processing core each time it is executed into the shared task results stored in the internal memory of the execution workstation.

[0040] Step S204: When a predetermined condition is met, the shared task result is used to reduce the task results of other execution workstations.

[0041] For step S201, the management workstation may be, for example, a driver, and the management workstation may play a central coordination role, scheduling tasks for each of the multiple processing cores, monitoring task progress, and the like.

[0042] For example, in Figure 1 In the system architecture shown, the processing core 103 may send an acquisition request for acquiring a task to the management workstation 101 , and receive a task from the management workstation 101 after the management workstation 101 allocates a task to the processing core 103 in response to the acquisition request.

[0043] For step S202, each processing core generates a task result after completing the execution of a task, and the task result has a predetermined data structure. The data structure can be predetermined according to a specific application, for example, it can be an array or other data structure defined by the user.

[0044] For step S203, the execution workstation includes multiple processing cores and an internal memory, and the internal memory stores a value called a shared task result. The shared task result is shared by multiple processing cores in the same execution workstation, that is, multiple processing cores can access and update the shared task result. The shared task result has the same data structure as the task result generated by the processing core each time it executes a task. Merging refers to the process of merging two or more data into one, for example, two or more data can be summed and other operations can be performed. In an embodiment of the present disclosure, merging the task result generated by each processing core each time it executes into the shared task result means merging the two task results into one task result, and using the result to update the current shared task result.

[0045] For example, an initial shared task result having the above-mentioned predetermined data structure may be generated in the internal memory first, and the initial shared task result may be initialized to 0. Then, after any one of the multiple processing cores completes a task, the obtained task result is merged with the current shared task result, and the current shared task result is updated with the merged result. That is, each processing core will update the shared task result after processing a task, so that the shared task result carries the information of multiple task results, but the amount of stored data is greatly reduced.

[0046] For another example, the task results generated for the first time when multiple processing cores process the tasks of the current task group can be stored in the internal memory as the initial shared task results. The task results generated by subsequent executions of the multiple processing cores are merged with the shared task results and the shared task results are updated, that is, the task result obtained after any one of the multiple processing cores completes the execution of a task is merged with the current shared task result, and the current shared task result is updated with the merged result. In an embodiment of the present disclosure, the current task group indicates the task group to which the task currently being processed belongs, and the task results in the same task group will be merged. The management executor can divide interrelated tasks into a task group, and obtain a final result that merges all task results after a task group is processed.

[0047] For step S204, when the predetermined condition is met, the execution workstation obtains the current final shared task result, which combines multiple task results, that is, carries information of multiple task results. In an embodiment of the present disclosure, the final shared task result can be used for further reduction with the task results of other execution workstations. "Reduction" refers to the process of merging multiple data stored in distributed storage into a single data.

[0048] The predetermined condition may be any predetermined condition for stopping the execution of the workstation local merge. For example, the predetermined condition may be that the shared task result has merged a predetermined number of task results, or the predetermined condition may be that an instruction from the management workstation is received.

[0049] In some embodiments of the present disclosure, an instruction refers to an instruction issued by the management workstation to determine that all tasks of the current task group have been completed based on the status of each task. For example, the management workstation can monitor the execution of tasks in each processing core, and each processing core sends the execution status of each task to the management workstation. In response to the execution status of the tasks of each processing core, it can be determined whether all tasks in the task group have been completed. When the management workstation determines that all tasks in the current task group have been completed, an instruction can be sent to the execution workstation. In response to the instruction issued from the management workstation, the execution workstation uses the current shared task result in its internal memory as the current final shared task result, and no longer uses the subsequent execution task results to update the shared task result.

[0050] For another example, the management workstation pre-sets the number of task results that each execution workstation should merge. For each execution workstation, after merging a predetermined number of task results, the current shared task result in the internal memory is used as the current final shared task result.

[0051] In some embodiments of the present disclosure, using a shared task result for reduction with task results of other execution workstations includes: serializing the shared task result and sending it to other execution workstations or management workstations for further reduction on the other execution workstations or management workstations; or receiving task results from other execution workstations and reducing them with the shared task results.

[0052] For example, the shared task results of multiple execution workstations can be further reduced and then sent to the management workstation. Specifically, for each execution workstation, the shared task results in the internal memory can be serialized and then sent to other execution workstations for reduction, or the shared task results can be received from other execution workstations and reduced. After the shared task results in each of the multiple execution workstations are further reduced, the further reduced results are serialized and sent to the management workstation.

[0053] In addition, in the data processing method provided by at least one embodiment of the present disclosure, when the predetermined condition is met, the execution workstation can also store the shared task result in a non-volatile storage device. Thus, when a failure occurs in the execution workstation, such as a process error, the shared task result can be obtained from the non-volatile storage device to achieve better fault tolerance. The non-volatile storage device can be any form of non-volatile storage device, such as a magnetic hard disk, a solid state hard disk, etc.

[0054] According to at least one embodiment of the present disclosure, if a task of an execution workstation fails to execute, the execution workstation can clear the current shared task result, and recalculate and regenerate the shared task result for the tasks of the current workgroup. In addition, when the management workstation detects that a task on an execution workstation fails to execute, all tasks of the current task group executed by the execution workstation can also be reallocated. Therefore, the embodiments of the present disclosure still support lineage fault tolerance.

[0055] In summary, according to the data processing method of the embodiment of the present disclosure, there is no need to independently store a task result, serialize the processing result, and send the processing result after each processing core completes each task, thereby saving storage, processing and communication overheads and improving system performance.

[0056] Figure 3 It shows that at least one embodiment of the present disclosure provides Figure 2 Flow chart of the method for step S203 in FIG.

[0057] like Figure 3 As shown, the method may include steps S301 to S302.

[0058] Step S301: Generate a shared task result having a predetermined data structure and an initial value of 0 in an internal memory.

[0059] Step S302: after any processing core completes executing a task and generates a task result, the generated task result is merged with the currently stored shared task result, and the shared task result is updated with the merged result.

[0060] Figure 4 It shows that at least one embodiment of the present disclosure provides Figure 2 Another method flow chart of step S203 in FIG.

[0061] like Figure 4 As shown, the method may include steps S401 to S402.

[0062] Step S401: storing the task results generated for the first time when multiple processing cores process the tasks of the current task group as initial shared task results.

[0063] Step S402: after any processing core completes executing a task and generates a task result, the generated task result is merged with the currently stored shared task result, and the shared task result is updated with the merged result.

[0064] Figure 3 and Figure 4The method for initializing the shared task result has been described above and will not be repeated here.

[0065] Figure 5 A schematic block diagram of an execution workstation 500 for distributed computing provided by at least one embodiment of the present disclosure is shown.

[0066] like Figure 5 As shown, the execution workstation 500 includes a receiving module 510 , a plurality of processing cores 520 , a merging module 530 and a result processing module 540 .

[0067] The receiving module 510 is configured to receive a task assigned to each of the plurality of processing cores from a management workstation.

[0068] The receiving module 510 may, for example, execute Figure 2 Described step S201.

[0069] Each processing core in the plurality of processing cores 520 is configured to execute the assigned tasks respectively, and generate a task result having a predetermined data structure after executing each task.

[0070] Each of the plurality of processing cores 520 may execute, for example, Figure 2 Described step S202.

[0071] The merging module 530 is configured to merge the task results generated by each processing core each time into the shared task results stored in the internal memory of the execution workstation, and the shared task results have the same data structure as the task results generated by each processing core each time.

[0072] The merge module 530 may, for example, execute Figure 2 Described step S203.

[0073] The result processing module 540 is configured to use the shared task result to reduce with the task results of other execution workstations when a predetermined condition is met.

[0074] The result processing module 540 may execute, for example, Figure 2 Described step S204.

[0075] For example, the receiving module 510, the multiple processing cores 520, the merging module 530 and the result processing module 540 can be implemented as hardware, software, firmware and any feasible combination thereof. For example, the receiving module 510, the multiple processing cores 520, the merging module 530 and the result processing module 540 can be dedicated or general circuits, chips or devices, etc., or can be a combination of a processor and a memory. The embodiments of the present disclosure do not limit the specific implementation forms of the above-mentioned modules.

[0076] It should be noted that in the embodiment of the present disclosure, the various modules of the execution workstation 500 for distributed computing correspond to the various steps of the aforementioned data processing method. For the specific functions of the execution workstation 500 for distributed computing, please refer to the relevant description of the data processing method, which will not be repeated here. Figure 5 The components and structures of the execution workstation 500 for distributed computing shown are merely exemplary and non-limiting. The execution workstation 500 for distributed computing may further include other components and structures as required.

[0077] At least one embodiment of the present disclosure also provides an electronic device, which includes a processor and a memory, and the memory includes one or more computer program modules. The one or more computer program modules are stored in the memory and are configured to be executed by the processor, and the one or more computer program modules include instructions for implementing the above-mentioned data processing method. The electronic device can merge the task results in the same execution workstation in the execution workstation before merging them with the task results in other execution workstations, thereby reducing storage, processing and communication overhead, thereby achieving better performance.

[0078] Figure 6 A schematic block diagram of an electronic device provided in some embodiments of the present disclosure. Figure 6 As shown, the electronic device 600 includes a processor 610 and a memory 620. The memory 620 is used to store non-transitory computer-readable instructions (e.g., one or more computer program modules). The processor 610 is used to run non-transitory computer-readable instructions, and when the non-transitory computer-readable instructions are run by the processor 610, one or more steps in the data processing method described above can be executed. The memory 620 and the processor 610 can be interconnected through a bus system and / or other forms of connection mechanisms (not shown).

[0079] For example, the processor 610 may be a central processing unit (CPU), a graphics processing unit (GPU), or other forms of processing units having data processing capabilities and / or program execution capabilities. For example, the central processing unit (CPU) may be an X86 or ARM architecture, etc. The processor 610 may be a general-purpose processor or a dedicated processor, and may control other components in the electronic device 600 to perform desired functions.

[0080] For example, the memory 620 may include any combination of one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disk read-only memory (CD-ROM), USB memory, flash memory, etc. One or more computer program modules may be stored on the computer-readable storage medium, and the processor 610 may run one or more computer program modules to implement various functions of the electronic device 600. Various applications and various data, as well as various data used and / or generated by the application, etc. may also be stored in the computer-readable storage medium.

[0081] It should be noted that in the embodiment of the present disclosure, the specific functions and technical effects of the electronic device 600 can refer to the above description of the data processing method, which will not be repeated here.

[0082] Figure 7 This is a schematic block diagram of another electronic device provided in some embodiments of the present disclosure. The electronic device 700 is suitable for implementing the data processing method provided in the embodiments of the present disclosure. The electronic device 700 may be a terminal device, etc. It should be noted that: Figure 7 The electronic device 700 shown is merely an example and does not impose any limitation on the functions and scope of use of the embodiments of the present disclosure.

[0083] like Figure 7 As shown, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 710, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 720 or a program loaded from a storage device 780 into a random access memory (RAM) 730. In the RAM 730, various programs and data required for the operation of the electronic device 700 are also stored. The processing device 710, the ROM 720, and the RAM 730 are connected to each other via a bus 740. An input / output (I / O) interface 750 is also connected to the bus 740.

[0084] Typically, the following devices may be connected to the I / O interface 750: an input device 760 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 770 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 780 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 790. The communication device 790 may allow the electronic device 700 to communicate with other electronic devices wirelessly or by wire to exchange data. Although Figure 7 The electronic device 700 is shown to have various devices, but it should be understood that it is not required to implement or possess all the devices shown, and the electronic device 700 may alternatively implement or possess more or fewer devices.

[0085] For example, according to an embodiment of the present disclosure, the above-mentioned data processing method can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the above-mentioned data processing method. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 790, or installed from the storage device 780, or installed from the ROM 720. When the computer program is executed by the processing device 710, the functions defined in the data processing method provided in the embodiment of the present disclosure can be implemented.

[0086] At least one embodiment of the present disclosure further provides a computer-readable storage medium, which is used to store non-transitory computer-readable instructions, and when the non-transitory computer-readable instructions are executed by a computer, the above-mentioned data processing method can be implemented. Using the computer-readable storage medium, task results in the same execution workstation can be merged in the execution workstation before being reduced with task results in other execution workstations, thereby reducing storage, processing and communication overhead.

[0087] Figure 8 A schematic diagram of a storage medium provided in some embodiments of the present disclosure. Figure 8 As shown, the storage medium 800 is used to store non-transitory computer-readable instructions 810. For example, when the non-transitory computer-readable instructions 810 are executed by a computer, one or more steps in the data processing method described above can be performed.

[0088] For example, the storage medium 800 may be applied to the electronic device 600. Figure 6 The memory 620 in the electronic device 600 is shown. For example, the description of the storage medium 800 can refer to Figure 6 The corresponding description of the memory 620 in the electronic device 600 is shown and will not be repeated here.

[0089] It should be noted that the drawings of the embodiments of the present disclosure only relate to the structures involved in the embodiments of the present disclosure, and other structures may refer to the general design. In the absence of conflict, the embodiments of the present disclosure and the features in the embodiments may be combined with each other to obtain new embodiments.

[0090] The above description is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto, and the protection scope of the present disclosure shall be based on the protection scope of the claims.

Claims

1. A data processing method for distributed computing performed by an execution workstation, wherein the execution workstation includes a plurality of processing cores, the method comprising: receiving, from a management workstation, a task assigned to each of the plurality of processing cores; Each of the plurality of processing cores executes the assigned tasks respectively, and generates a task result having a predetermined data structure after executing each task; Merging the task result generated by each processing core each time it is executed into the shared task result stored in the internal memory of the execution workstation, wherein the shared task result and the task result generated by each processing core each time it is executed have the same data structure, and the multiple processing cores can access and update the shared task result, and when merging the task result generated by each processing core each time it is executed into the shared task result, merge the task result with the currently stored shared task result, and use the merged result to update the currently stored shared task result; as well as When a predetermined condition is met, the shared task result is used to reduce the task results of other execution workstations.

2. The data processing method according to claim 1, wherein The predetermined condition is that the shared task result has merged a predetermined number of task results.

3. The data processing method according to claim 1, wherein The predetermined condition is receiving an instruction from the management workstation.

4. The data processing method according to claim 3, wherein The method further comprises sending the status of each task being completed to the management workstation; The instruction is issued by the management workstation after determining that all tasks of the current task group have been completed according to the status of each task.

5. The data processing method according to claim 1, wherein the step of merging the task results generated by each processing core each time into the shared task results stored in the internal memory of the execution workstation comprises: Generating a shared task result having the predetermined data structure and an initial value of 0 in the internal memory; as well as After any processing core completes executing a task and generates a task result, the generated task result is merged with the currently stored shared task result, and the shared task result is updated with the merged result.

6. The data processing method according to claim 1, wherein the step of merging the task results generated by each processing core each time into the shared task results stored in the internal memory of the execution workstation comprises: storing the task results generated for the first time when the multiple processing cores process the tasks of the current task group as initial shared task results; as well as After any processing core completes executing a task and generates a task result, the generated task result is merged with the currently stored shared task result, and the shared task result is updated with the merged result.

7. The data processing method according to any one of claims 1 to 6, wherein the using the shared task result to reduce with the task results of other execution workstations comprises: Serializing the shared task results and sending them to other execution workstations or the management workstation; or Task results are received from other execution workstations and reduced with the shared task results.

8. The data processing method according to any one of claims 1 to 6, further comprising: When the predetermined condition is met, the shared task result is stored in a non-volatile storage device.

9. An execution workstation for distributed computing, the execution workstation comprising a receiving module, a plurality of processing cores, a merging module, an internal memory and a result processing module, wherein: The receiving module is used to receive a task assigned to each of the multiple processing cores from a management workstation; Each of the plurality of processing cores is used to execute the assigned tasks respectively, and generate a task result having a predetermined data structure after executing each task; The merging module is used to merge the task result generated by each processing core each time it is executed into the shared task result stored in the internal memory of the execution workstation, wherein the shared task result and the task result generated by each processing core each time it is executed have the same data structure, and the multiple processing cores can access and update the shared task result. When merging the task result generated by each processing core each time it is executed into the shared task result, the task result is merged with the currently stored shared task result, and the merged result is used to update the currently stored shared task result; and The result processing module is used to reduce the shared task result with the task results of other execution workstations when a predetermined condition is met.

10. An electronic device comprising a processor and a memory, wherein the memory stores one or more computer program instructions, wherein: The one or more computer program instructions are stored in the memory and, when executed by the processor, implement the steps of the data processing method according to any one of claims 1 to 8.

11. A computer-readable storage medium non-temporarily storing computer-readable instructions, wherein: When the computer-readable instructions are executed by a processor, the steps of the data processing method according to any one of claims 1 to 8 are implemented.