Data processing method and device, electronic equipment and storage medium

By using shared memory pools and hardware acceleration units in Hadoop for sorting and reducing operations of mapped output data, the problems of CPU resource consumption and data transmission overhead in traditional MapReduce tasks are solved, and the performance of big data processing is improved.

CN119003208AInactive Publication Date: 2024-11-22CHINA TELECOM CORP LTD TECHNOLOGY INNOVATION CENTER +1

Patent Information

Application Number
CN202411067823.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-05
Publication Date
2024-11-22
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The mapping sorting MapReduce task of traditional Hadoop consumes a large amount of CPU resources and generates network and disk IO overhead during inter-process data transmission, becoming a performance bottleneck.

Method used

By writing the output data of each mapping task to the shared memory pool for sorting and reduction operations, the hardware acceleration unit is used for task calculations, reducing network and disk data transmission overhead.

Benefits of technology

It effectively reduces the network and disk data transmission overhead caused by CPU mapping sorting, and improves the performance of big data processing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119003208A_ABST
    Figure CN119003208A_ABST
Patent Text Reader

Abstract

The invention provides a data processing method and device, electronic equipment and a storage medium, and relates to the technical field of big data distributed computing. The method comprises the following steps: writing mapping output data of each mapping task into a shared memory pool; sequencing the mapping output data of each mapping task in a shared memory pool, and determining a plurality of sequenced output data; and performing reduction operation on the plurality of sorted output data to determine merged data. According to the method, each piece of mapping output data is written into the shared memory pool, and the mapping output data is uniformly shared through the shared memory pool, so that the network and disk data transmission overhead in the sorting operation stage is reduced, and the problem that the network and disk data transmission overhead is large due to the fact that a CPU executes mapping sorting is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0002] Traditional Hadoop MapReduce includes Map, Reduce and Shuffle tasks, which are performed by the CPU and handed over to SSD or disk for final and temporary data storage. However, these tasks consume a lot of CPU resources, and Shuffle generates a lot of network overhead during data transmission between processes, and also generates a lot of disk IO for storing intermediate data, which will become a performance bottleneck under the MapReduce architecture.

[0003] In the existing technology, the resource utilization of the system is improved by advancing the execution time of the Reduce process and balancing the resource usage gap between Map and Reduce tasks. However, this is only a coarse-grained management, which is only a drop in the bucket for processing big data and does not solve the fundamental problem.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The present disclosure provides a data processing method, device, electronic device and storage medium, which at least to some extent overcome the problem of high network and disk data transmission overhead caused by CPU executing mapping sorting in the related art.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0007] According to one aspect of the present disclosure, a data processing method is provided, including: writing mapping output data of each mapping task into a shared memory pool; performing a sorting operation on the mapping output data of each mapping task in the shared memory pool to determine a plurality of sorted output data; performing a reduction operation on the plurality of sorted output data to determine merged data.

[0008] In some embodiments, before writing the mapping output data of each mapping task into the shared memory pool, the method further includes: acquiring a mapping sorting task; and performing task calculation on the mapping sorting task by a plurality of first hardware acceleration units to determine the mapping output data of each mapping task.

[0009] In some embodiments, before the multiple first hardware acceleration units perform task calculation on the mapping sorting tasks and determine the mapping output data of each mapping task, the method further includes: the CPU side parses the mapping sorting tasks and selects a first hardware acceleration unit to start task calculation.

[0010] In some embodiments, the method further includes: defining an operation logic of the first hardware acceleration unit according to a mapping task, and determining the first hardware acceleration unit.

[0011] In some embodiments, performing a reduction operation on the plurality of sorted output data to determine the merged data includes: performing a reduction operation on the plurality of sorted output data by a plurality of second hardware acceleration units to determine the merged data.

[0012] In some embodiments, the method further includes: defining an operation logic of the second hardware acceleration unit according to the sorting task, and determining the second hardware acceleration unit.

[0013] In some embodiments, the shared memory pool is a compute fast link shared pool.

[0014] According to another aspect of the present disclosure, a data processing device is also provided, including: a shared memory pool writing module, used to write the mapping output data of each mapping task into a shared memory pool; a sorting operation module, used to perform a sorting operation on the mapping output data of each mapping task in the shared memory pool to determine a plurality of sorted output data; and a reduction operation module, used to perform a reduction operation on the plurality of sorted output data to determine merged data.

[0015] According to another aspect of the present disclosure, an electronic device is also provided, which includes: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned data processing methods by executing the executable instructions.

[0016] In some embodiments, the data processing device further includes a first acceleration module for acquiring mapping sorting tasks; a plurality of first hardware acceleration units perform task calculations on the mapping sorting tasks to determine mapping output data of each mapping task.

[0017] In some embodiments, the data processing device further includes an acceleration selection module, which is used for the CPU to parse the mapping sorting task and select the first hardware acceleration unit to start task calculation.

[0018] In some embodiments, the first acceleration module is further used to define the operation logic of the first hardware acceleration unit according to the mapping task and determine the first hardware acceleration unit.

[0019] In some embodiments, the reduction operation module is further used for multiple second hardware acceleration units to perform reduction operations on multiple sorted output data to determine merged data.

[0020] In some embodiments, the reduction operation module is further used to define the operation logic of the second hardware acceleration unit according to the sorting task and determine the second hardware acceleration unit.

[0021] In some embodiments, the shared memory pool in the above data processing device is a computing fast link shared pool.

[0022] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the data processing method described in any one of the above is implemented.

[0023] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, any one of the above-mentioned data processing methods is implemented.

[0024] The data processing method provided in the embodiments of the present disclosure writes the mapping output data of each mapping task into a shared memory pool; performs a sorting operation on the mapping output data of each mapping task in the shared memory pool to determine a plurality of sorted output data; performs a reduction operation on the plurality of sorted output data to determine the merged data. The present application writes each mapping output data into a shared memory pool, and uniformly shares the mapping output data through the shared memory pool, thereby reducing the network and disk data transmission overhead in the sorting operation stage, and solves the problem that the CPU performs mapping sorting, resulting in large network and disk data transmission overhead.

[0025] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0027] Figure 1 A schematic diagram showing a data processing system structure in an embodiment of the present disclosure;

[0028] Figure 2 A flow chart of a data processing method in an embodiment of the present disclosure is shown;

[0029] Figure 3 A flowchart showing a specific example of a data processing method in an embodiment of the present disclosure;

[0030] Figure 4 A flowchart showing another specific example of a data processing method in an embodiment of the present disclosure;

[0031] Figure 5 A flowchart showing another specific example of a data processing method in an embodiment of the present disclosure;

[0032] Figure 6 A schematic diagram of a data processing device in an embodiment of the present disclosure is shown;

[0033] Figure 7 A structural block diagram of a computer device according to an embodiment of the present disclosure is shown;

[0034] Figure 8 A schematic diagram showing a computer-readable storage medium in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0036] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0037] For ease of understanding, before introducing the embodiments of the present disclosure, several terms involved in the embodiments of the present disclosure are first explained as follows:

[0038] Hadoop: Hadoop is a distributed system infrastructure developed by the Apache Foundation;

[0039] MR: MapReduce, map reduction;

[0040] CXL: Compute Express Link, an open interconnect protocol:

[0041] DPU: Data Processing Unit, data processor;

[0042] FPGA: Field Programmable Gate Array, field programmable logic gate array;

[0043] ASIC: Application Specific Integrated Circuit, ASIC chip.

[0044] The specific implementation of the embodiment of the present disclosure is described in detail below with reference to the accompanying drawings.

[0045] Figure 1 FIG. 1 shows an exemplary application system architecture diagram to which the data processing method in the embodiment of the present disclosure can be applied. Figure 1 As shown, the system architecture may include a computing cluster 101 , a network 102 , and a shared memory pool 103 .

[0046] The network 102 is a medium for providing a communication link between the terminal device 101 and the server 103, and can be a wired network or a wireless network.

[0047] Optionally, the wireless network or wired network described above uses standard communication technology and / or protocol. The network is usually the Internet, but it can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a dedicated network or any combination of a virtual private network). In some embodiments, the data exchanged through the network is represented by technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPSec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above data communication technologies.

[0048] The computing cluster 101 may be a computing cluster based on MapReduce. MapReduce is a distributed parallel computing framework applied to large-scale data sets. The computing framework refers to the structure of the computing process or flow that realizes a task or a work from the beginning to the end, and the distributed parallel computing framework further refers to splitting a large task into multiple small tasks and distributing them to multiple computing devices (computing devices, computing devices, computing devices), and multiple computing devices perform calculations simultaneously. The MapReduce computing framework emphasizes the parallel computing idea of ​​"divide and conquer, divide first and then combine". It splits a large and complex task into multiple small tasks, distributes them to each computing device for calculation, and then merges the calculation results. The MapReduce computing framework mainly includes Map operations and Reduce operations. The Map operation is also called a mapping operation, which is to split the data and decompose the complex task into several "simple tasks" for parallel processing. The Reduce operation is to summarize the data, that is, to globally summarize the results of the Map operation.

[0049] In an instance, each node in the computing cluster represents a server or a computing node, and the computing unit is a component or part of it. Generally speaking, a computing node can have one or more computing units. The specific configuration varies from product to product. The CPU of the local computing node will select which computing unit to schedule to work, and there will be a corresponding scheduling algorithm inside.

[0050] The shared memory pool 103 may be a large-capacity memory that can be accessed by different central processing units (CPUs), so that multiple processes can access the same memory space.

[0051] Optionally, the shared memory pool may be established through CXL, DPU, memory virtualization, or memory channel technology.

[0052] In a specific example, when the system is running, the host side sends a MapReduce task scheduling instruction; the Map hardware acceleration unit is called to perform mapper task calculations and store intermediate results in a shared memory pool; the Shuffle task reads data on the shared memory pool for operation; and Reduce reads the Shuffled data and performs a Reduce operation.

[0053] Those skilled in the art will know that Figure 1 The number of computing devices and networks in the example is only illustrative, and any number of computing devices and networks may be provided according to actual needs. This disclosure embodiment does not limit this.

[0054] Figure 2 A flow chart of a data processing method in an embodiment of the present disclosure is shown as follows: Figure 2As shown, the data processing method provided in the embodiment of the present disclosure includes the following steps:

[0055] S202, writing the mapping output data of each mapping task into a shared memory pool.

[0056] It should be noted that the above-mentioned mapping task can be a Map task. Specifically, the Map task is the first stage in the MapReduce programming model, which is responsible for processing part of the input data. It divides the input data into key-value pairs, and applies the user-defined Map function to process these key-value pairs, and generates intermediate key-value pairs as output. For example, there is an input data set containing a large number of text files, each file containing a series of lines. The Map task can read these files, convert each line of text into a key-value pair (for example, the key can be the line number and the value can be the line text), and apply the Map function to process these key-value pairs.

[0057] The above mapping output data can be the intermediate key-value pairs of the Map task.

[0058] The above-mentioned shared memory pool may be established based on CXL, DPU, memory virtualization, and memory channel technology.

[0059] S204: performing a sorting operation on the mapping output data of each mapping task in the shared memory pool to determine a plurality of sorted output data.

[0060] It should be noted that the above-mentioned sorting operation can be Shuffle. Specifically, the Shuffle task is an important step in the MapReduce process, which occurs between the Map stage and the Reduce stage. It is responsible for sorting and grouping the output of the Map task according to the key, and sending the sorted key-value pairs to the corresponding Reduce task. For example, the intermediate key-value pairs generated by the Map task are unordered. The Shuffle task will sort and group these key-value pairs to ensure that the key-value pairs with the same key (equivalent to the above-mentioned sorted output data) are gathered together and sent to the Reduce task that processes the key, so that the Reduce task can efficiently process the key-value pairs with the same key and perform the reduction operation.

[0061] S206, performing a reduction operation on the plurality of sorted output data to determine merged data.

[0062] It should be noted that the above reduction operation can be Reduce. Specifically, the Reduce task is the second stage in the MapReduce programming model, which is responsible for receiving the intermediate key-value pairs from the Map task, and applying the user-defined Reduce function to these key-value pairs to perform reduction operations, and finally output the final result. For example, the Map task has generated a series of key-value pairs, where the key is a word and the value is the number of times the word appears. The Reduce task will receive these key-value pairs and aggregate the key-value pairs with the same key. Then, the Reduce function can sum these values ​​to calculate the total number of times each word appears in the entire data set.

[0063] For example, user A sends a MapReduce task to the cluster, the cluster scheduler sends the request to node B, node B offloads the Map task to the Map computing engine of the hardware acceleration unit, the Map computing engine writes the intermediate results of the Map into the shared memory pool, and the Reduce task is similarly offloaded to the Reduce computing engine of the hardware acceleration unit. The Reduce task obtains the Map data from the shared resource pool (equivalent to the above-mentioned shared memory pool), performs Shuffle calculations, obtains the final results and stores them.

[0064] The present application writes each mapping output data into a shared memory pool, and shares the mapping output data (intermediate data results) uniformly through the shared memory pool, thereby reducing the network and disk data transmission overhead in the Shuffle phase (sorting operation phase), and solving the problem of high network and disk data transmission overhead caused by CPU executing mapping sorting.

[0065] In one embodiment of the present disclosure, Figure 3 As shown, the data processing method provided in the embodiment of the present disclosure can determine the mapping output data by the following steps before writing the mapping output data of each mapping task into the shared memory pool, thereby offloading the Map task on the host side and reducing the CPU computing pressure:

[0066] S302, obtaining a mapping sorting task;

[0067] S304: The plurality of first hardware acceleration units perform task calculation on the mapping sorting tasks to determine mapping output data of each mapping task.

[0068] It should be noted that the above-mentioned mapping and sorting tasks can be MapReduce tasks. MapReduce tasks include: Map tasks, Reduce tasks and Shuffle tasks. The Map task is responsible for processing input data and generating intermediate results, the Reduce task is responsible for reducing intermediate results and outputting final results, and the Shuffle task is responsible for transferring and sorting data between Map and Reduce tasks. These tasks work together to implement the core functions of the MapReduce programming model. The first hardware acceleration unit can be a hardware chip, for example, a hardware acceleration unit implemented by FPGA, ASIC, etc., to unload things that usually require the CPU to complete through hardware, thereby accelerating and reducing CPU overhead.

[0069] In one embodiment of the present disclosure, Figure 4 As shown, the data processing method provided in the embodiment of the present disclosure can select the first hardware acceleration unit through the following steps before multiple first hardware acceleration units perform task calculation on the mapping sorting task and determine the mapping output data of each mapping task, and the CPU of the local computing node can select and schedule the first hardware acceleration unit to realize intelligent scheduling:

[0070] S402: The CPU parses the mapping sorting task and selects the first hardware acceleration unit to start task calculation.

[0071] In one example of the present disclosure, the first hardware acceleration unit is determined according to the operation logic of the first hardware acceleration unit defined by the mapping task.

[0072] It should be noted that the first hardware acceleration unit is programmable, and the operation logic of the hardware acceleration unit can be defined according to the type of task that the user wants to offload, for example, it can be implemented according to the calculation logic of the Map task.

[0073] In one embodiment of the present disclosure, Figure 5 As shown, the data processing method provided in the embodiment of the present disclosure can determine the merged data through the following steps, which can offload the Reduce task on the host side and reduce the CPU computing pressure:

[0074] S502: Multiple second hardware acceleration units perform a reduction operation on multiple sorted output data to determine merged data.

[0075] In one example of the present disclosure, the second hardware acceleration unit is determined according to the operation logic of the second hardware acceleration unit defined by the sequencing task.

[0076] It should be noted that the second hardware acceleration unit is programmable, and the operation logic of the hardware acceleration unit can be defined according to the type of task that the user wants to offload, for example, it can be implemented according to the calculation logic of the Reduce task.

[0077] In one embodiment of the present disclosure, the shared memory pool is a computing fast link shared pool.

[0078] It should be noted that the CXL compute fast link shared pool enables high-speed and efficient interconnection between the CPU and GPU, FPGA or other accelerators, while supporting unified memory access and memory pooling, thus meeting the requirements of high-performance heterogeneous computing.

[0079] The present invention combines a shared memory pool with hardware acceleration and applies it to MapReduce tasks. The hardware acceleration unit is used to offload the Map and Reduce tasks on the host side. At the same time, a shared memory pool is introduced to share the intermediate data results uniformly through the shared memory pool, thereby reducing the network and disk data transmission overhead. The combination of the two solves the problem that the CPU performs mapping sorting and causes high network and disk data transmission overhead.

[0080] Based on the same inventive concept, the present disclosure also provides a data processing device in the following embodiments. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0081] Figure 6 A schematic diagram of a data processing device in an embodiment of the present disclosure is shown. Figure 6 As shown, the device includes: a shared memory pool writing module 61, a sorting operation module 62 and a reduction operation module 63.

[0082] The shared memory pool writing module 61 is used to write the mapping output data of each mapping task into the shared memory pool;

[0083] A sorting operation module 62, used for performing a sorting operation on the mapping output data of each mapping task in the shared memory pool to determine a plurality of sorted output data;

[0084] The reduction operation module 63 is used to perform a reduction operation on the plurality of sorted output data to determine the merged data.

[0085] In an example of the present disclosure, the data processing device further includes a first acceleration module for acquiring mapping sorting tasks; a plurality of first hardware acceleration units perform task calculations on the mapping sorting tasks to determine mapping output data of each mapping task.

[0086] In an example of the present disclosure, the data processing device further includes an acceleration selection module, which is used for the CPU to parse the mapping sorting task and select the first hardware acceleration unit to start task calculation.

[0087] In an example of the present disclosure, the first acceleration module is further used to define the operation logic of the first hardware acceleration unit according to the mapping task and determine the first hardware acceleration unit.

[0088] In an example of the present disclosure, the reduction operation module is further used for multiple second hardware acceleration units to perform reduction operations on multiple sorted output data to determine merged data.

[0089] In an example of the present disclosure, the reduction operation module is further used to define the operation logic of the second hardware acceleration unit according to the sorting task and determine the second hardware acceleration unit.

[0090] In an example of the present disclosure, the shared memory pool in the above-mentioned data processing device is a computing fast link shared pool.

[0091] It should be noted that the shared memory pool writing module 61, the sorting operation module 62 and the reduction operation module 63 correspond to S202 to S204 in the method embodiment, and the examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above method embodiment. It should be noted that the above modules as part of the device can be executed in a computer system such as a set of computer executable instructions.

[0092] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0093] Refer to the following Figure 7 An electronic device 700 according to this embodiment of the present disclosure is described. Figure 7 The electronic device 700 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0094] like Figure 7 As shown, the electronic device 700 is in the form of a general computing device. The components of the electronic device 700 may include but are not limited to: at least one processing unit 710, at least one storage unit 720, and a bus 730 connecting different system components (including the storage unit 720 and the processing unit 710).

[0095] The storage unit stores program codes, which can be executed by the processing unit 710, so that the processing unit 710 executes the steps described in the above “exemplary method” section of this specification according to various exemplary embodiments of the present disclosure.

[0096] For example, the processing unit 710 can execute the following steps of the above method embodiment: write the mapping output data of each mapping task into a shared memory pool; perform a sorting operation on the mapping output data of each mapping task in the shared memory pool to determine multiple sorted output data; perform a reduction operation on the multiple sorted output data to determine merged data.

[0097] For example, the processing unit 710 may execute the following steps of the above method embodiment: obtaining a mapping sorting task; and performing task calculation on the mapping sorting task by a plurality of first hardware acceleration units to determine mapping output data of each mapping task.

[0098] For example, the processing unit 710 may execute the following steps of the above method embodiment: the CPU side parses the mapping sorting task and selects the first hardware acceleration unit to start task calculation.

[0099] For example, the processing unit 710 may perform the following steps of the above method embodiment: define the operation logic of the first hardware acceleration unit according to the mapping task, and determine the first hardware acceleration unit.

[0100] For example, the processing unit 710 may execute the following steps of the above method embodiment: a plurality of second hardware acceleration units perform a reduction operation on a plurality of sorted output data to determine merged data.

[0101] For example, the processing unit 710 may perform the following steps of the above method embodiment: define the operation logic of the second hardware acceleration unit according to the sorting task, and determine the second hardware acceleration unit.

[0102] The storage unit 720 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 7201 and / or a cache storage unit 7202 , and may further include a read-only storage unit (ROM) 7203 .

[0103] The storage unit 720 may also include a program / utility 7204 having a set (at least one) of program modules 7205, such program modules 7205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0104] Bus 730 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0105] The electronic device 700 may also communicate with one or more external devices 740 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 700, and / or communicate with any device that enables the electronic device 700 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 750. Furthermore, the electronic device 700 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 760. As shown, the network adapter 760 communicates with other modules of the electronic device 700 via a bus 730. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0106] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0107] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart may be implemented as a computer program product, which includes: a computer program, which implements the above data processing method when executed by a processor.

[0108] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is further provided, which may be a readable signal medium or a readable storage medium. Figure 8 A schematic diagram of a computer-readable storage medium in an embodiment of the present disclosure is shown. Figure 8As shown, a program product capable of implementing the above method of the present disclosure is stored on the computer-readable storage medium 800. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary implementations of the present disclosure described in the above “Exemplary Method” section of this specification.

[0109] For example, when the program product in the embodiment of the present disclosure is executed by a processor, the method of implementing the following steps is implemented:

[0110] Write the mapping output data of each mapping task into the shared memory pool;

[0111] Sorting the mapping output data of each mapping task in the shared memory pool to determine a plurality of sorted output data;

[0112] Perform reduction operations on multiple sorted output data to determine merged data.

[0113] For example, when the program product in the embodiment of the present disclosure is executed by a processor, the method of implementing the following steps is implemented:

[0114] Acquire mapping sorting tasks; multiple first hardware acceleration units perform task calculations on the mapping sorting tasks to determine mapping output data of each mapping task.

[0115] For example, when the program product in the embodiment of the present disclosure is executed by a processor, the method of implementing the following steps is implemented:

[0116] The CPU side parses the mapping sorting task and selects the first hardware acceleration unit to start task calculation.

[0117] For example, when the program product in the embodiment of the present disclosure is executed by a processor, the method of implementing the following steps is implemented:

[0118] The operation logic of the first hardware acceleration unit is defined according to the mapping task, and the first hardware acceleration unit is determined.

[0119] For example, when the program product in the embodiment of the present disclosure is executed by a processor, the method of implementing the following steps is implemented:

[0120] The plurality of second hardware acceleration units perform reduction operations on the plurality of sorted output data to determine merged data.

[0121] For example, when the program product in the embodiment of the present disclosure is executed by a processor, the method of implementing the following steps is implemented:

[0122] The operation logic of the second hardware acceleration unit is defined according to the sorting task, and the second hardware acceleration unit is determined.

[0123] More specific examples of computer-readable storage media in the present disclosure may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0124] In the present disclosure, a computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0125] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0126] In a specific implementation, the program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0127] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0128] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0129] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0130] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A data processing method, characterized in that: include: Write the mapping output data of each mapping task into the shared memory pool; performing a sorting operation on the mapping output data of each mapping task in the shared memory pool to determine a plurality of sorted output data; The plurality of sorted output data are subjected to a reduction operation to determine merged data.

2. The data processing method according to claim 1, characterized in that: Before writing the mapping output data of each mapping task into the shared memory pool, the method further comprises: Get the mapping sorting task; A plurality of first hardware acceleration units perform task calculation on the mapping sorting tasks to determine mapping output data of each mapping task.

3. The data processing method according to claim 2, characterized in that: Before the plurality of first hardware acceleration units perform task calculation on the mapping sorting tasks to determine mapping output data of each mapping task, the method further includes: The CPU side parses the mapping sorting task and selects the first hardware acceleration unit to start task calculation.

4. The data processing method according to claim 2, characterized in that: The method further comprises: The operation logic of the first hardware acceleration unit is defined according to the mapping task, and the first hardware acceleration unit is determined.

5. The data processing method according to claim 1, characterized in that: Performing a reduction operation on the plurality of sorted output data to determine the merged data includes: The plurality of second hardware acceleration units perform reduction operations on the plurality of sorted output data to determine merged data.

6. The data processing method according to claim 5, characterized in that: The method further comprises: The operation logic of the second hardware acceleration unit is defined according to the sorting task, and the second hardware acceleration unit is determined.

7. The data processing method according to any one of claims 1 to 5, characterized in that: The shared memory pool is a computing fast link shared pool.

8. A data processing device, characterized in that: include: A shared memory pool writing module is used to write the mapping output data of each mapping task into the shared memory pool; A sorting operation module, used for performing a sorting operation on the mapping output data of each mapping task in the shared memory pool to determine a plurality of sorted output data; The reduction operation module is used to perform a reduction operation on the plurality of sorted output data to determine merged data.

9. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the data processing method according to any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data processing method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Data storage system

    CN107066205A

  • MapReduce-based K-means clustering algorithm FPGA acceleration system

    CN107368375A

  • Data processing method in distributed system and related system

    CN115809015A

  • In-memory data shuffling

    WO2017086987A1

  • Data processing method, device, and system

    WO2017113277A1

Cited By

  • Data sharing method and device, electronic equipment and storage medium

    CN120723846A