Method, device and equipment for realizing parallel computation based on central processing unit and medium

Abstract

The application relates to the technical field of computers and discloses a method, device and equipment for realizing parallel computing based on a central processing unit and a medium, the method comprising the following steps: obtaining a context file and a to-be-processed file, and performing fragmentation on the to-be-processed file according to the context file to obtain a first preset number of file fragments; obtaining a mapping kernel function, a grouping function and a merging kernel function; generating a first preset number of mapping tasks according to the file fragments and the mapping kernel function, performing parallel computing on the mapping tasks by using an operation unit to obtain a processing result; grouping the processing result by using the grouping function to obtain a grouping result; generating a second preset number of merging tasks according to the merging kernel function and the grouping result, and processing the merging tasks by using the operation unit to obtain a parallel computing result of the to-be-processed file. The application solves the problem that parallel computing is too dependent on a graphics processing unit and cannot be performed by using a central processing unit to realize high operation efficiency.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically to a method, apparatus, device, and medium for implementing parallel computing based on a central processing unit. Background Technology

[0002] With the rapid development of new-generation information technology, the application of big data computing, artificial intelligence technology, and the increasing demand for digital and intelligent transformation have had a positive impact on various industries. Artificial intelligence (AI) algorithms and high-performance computing (HPC) are the cornerstones of new-generation information technology, and both AI algorithms and HPC require parallel computing.

[0003] Currently, parallel computing technologies rely on graphics processing units (GPUs) to efficiently process massive amounts of data. Additionally, OpenCL (Open Computing Language) and distributed system infrastructure (Hadoop) are used. However, when using only a central processing unit (CPU) for parallel computing, Hadoop's CPU adaptability is limited and cannot meet general requirements. OpenCL technology, by simply changing the CPU's serial processing logic to parallelizable kernel functions, offers limited improvement in the computing power of CPU-based parallel computing.

[0004] Therefore, existing technologies suffer from the problem that parallel computing relies too heavily on graphics processing units (GPUs) and cannot perform high-efficiency parallel computing using only the central processing unit (CPU). Summary of the Invention

[0005] In view of this, the present invention provides a method, apparatus, device and medium for parallel computing based on a central processing unit, in order to solve the problem that parallel computing relies too much on a graphics processing unit and cannot perform high-efficiency parallel computing solely through a central processing unit.

[0006] In a first aspect, the present invention provides a method for implementing parallel computing based on a central processing unit, the method comprising:

[0007] The context file and the file to be processed are obtained, and the file to be processed is segmented according to the context file to obtain a first preset number of file segments. The context file is used to determine the processing unit, and the processing unit is contained in the central processing unit.

[0008] Obtain the mapping kernel function, grouping function, and merging kernel function, which are generated based on a preset technical framework;

[0009] The first preset number of mapping tasks are generated based on file fragmentation and mapping kernel function. The mapping tasks are then computed in parallel using the computing unit to obtain the processing results.

[0010] The processing results are grouped using a grouping function to obtain the grouped results.

[0011] Based on the merging kernel function and the grouping results, a second preset number of merging tasks are generated. The merging tasks are then processed using the computing unit to obtain the parallel computing results of the files to be processed.

[0012] This embodiment provides a method for parallel computing based on a central processing unit (CPU). The file to be processed is fragmented. Based on the mapping kernel function and the file fragments, mapping tasks are generated and delivered to the CPU's processing unit for parallel computation. The processing results are grouped, and finally, based on the merging kernel function and the grouping results, merging tasks are generated and delivered to the CPU's processing unit for merging, yielding the parallel computation result. This method utilizes only the CPU for parallel computing, making full use of the CPU's limited computing resources. Without GPU assistance, it maximizes computing power to meet the demands of parallel computing. It solves the problem in related technologies where parallel computing relies too heavily on the graphics processing unit (GPU), making it impossible to perform high-efficiency parallel computing solely through the CPU.

[0013] In an alternative implementation, before obtaining the context file and the file to be processed, the method further includes:

[0014] Obtain a third preset number of central processing units, parameter information of the central processing units, and platform information of the preset technical framework. The parameter information is used to determine the core of the central processing unit, and the platform information is the information required to implement the preset technical framework.

[0015] Based on platform information and parameter information, the central processing unit is meshed using a preset technical framework to obtain a fourth preset number of processing units, where each processing unit corresponds to a core of the central processing unit.

[0016] A context file is generated based on the fourth preset quantity, parameter information, and platform information.

[0017] In this embodiment, a pre-defined technical framework is used to perform grid-based processing on the central processing unit, generating multiple computing units. These computing units can process tasks simultaneously, and the parallel processing capability of the present invention is significantly improved through the use of multiple computing units.

[0018] In one optional implementation, the mapping task is performed in parallel using a computing unit to obtain the processing result, including:

[0019] The processing unit obtains the data mapping business logic and file fragmentation based on the mapping task;

[0020] The processing unit generates a fifth preset number of key-value pairs based on the data mapping business logic and file fragmentation;

[0021] The key-value pairs of all operation units are summarized to obtain the processing result.

[0022] In this embodiment, the computing unit performs parallel computation on the mapping task to obtain the processing result, which improves the efficiency of the present invention in processing the file to be processed; the data mapping business logic is used to instruct the computing unit on how to calculate, which improves the accuracy and stability of the calculation.

[0023] In one optional implementation, the processing results are grouped using a grouping function to obtain grouped results, including:

[0024] The processing results yield key-value pairs, where each key-value pair contains a key;

[0025] Based on the key, the key-value pairs are grouped to obtain the grouping results. The grouping results contain a second preset number of key-value pair groups, and each key-value pair group contains a sixth preset number of key-value pairs with the same key.

[0026] The grouping results are passed to the merge kernel function.

[0027] In this implementation, a grouping function is used to group key-value pairs with the same key into the same key-value pair group, which facilitates the subsequent merging of processing results, simplifies the merging business logic in the subsequent merging process, avoids merging errors, and prevents them from affecting the final parallel computing results.

[0028] In one optional implementation, a second preset number of merging tasks are generated based on the merging kernel function and the grouping results. The merging tasks are then processed using a computing unit to obtain the parallel computation results of the file to be processed, including:

[0029] Based on the grouping results, a second preset number of key-value pair groups are obtained;

[0030] Based on the merge kernel function and key-value pair grouping, a second preset number of merge tasks are generated, wherein the merge kernel function is used to write the data merging business logic into the merge task;

[0031] When processing the merging task, the processing unit merges the values ​​of key-value pairs in the key-value pair group according to the data merging business logic, and obtains the merging result of each key-value pair group.

[0032] The results of merging all key-value pairs are summarized to obtain the parallel computation results.

[0033] In this implementation, generating merge tasks assigns each merge task to a processing unit, improving merge efficiency. Utilizing data merging business logic instructs the processing unit on how to merge, enhancing the accuracy and stability of the merge process.

[0034] In one optional implementation, the file to be processed is fragmented according to the context file to obtain a first preset number of file fragments, including:

[0035] The server resource threshold, the fourth preset number of computing units, and the memory capacity threshold are obtained from the context file.

[0036] Get the file size of the file to be processed;

[0037] If the file size is equal to or less than the memory capacity threshold, the file fragment size is obtained based on the fourth preset quantity, the file size, and the server resource threshold.

[0038] Based on the file fragment size and the file to be processed, a first preset number of file fragments are obtained.

[0039] In this embodiment, the file fragment size is first determined, and the file to be processed is fragmented according to the file fragment size to obtain a first preset number of file fragments. This allows different processing units to process the file to be processed simultaneously, and only one file fragment needs to be processed, thus improving the computational efficiency of parallel computing.

[0040] In one optional implementation, after obtaining the parallel computation results of the file to be processed, the method further includes:

[0041] Obtain the actual business requirements, and obtain the business logic implementation functions based on the actual business requirements;

[0042] Based on the business logic implementation function, write the parallel computing results into the business logic implementation path and / or output the parallel computing results.

[0043] In this embodiment, the business logic implementation function is obtained according to actual business needs, executed, and the parallel computing results are written into the business logic implementation path and / or output. This enables the invention to have more functions and expands its scope of application.

[0044] In an alternative implementation, before obtaining the mapping kernel function, the grouping function, and the merging kernel function, the method further includes:

[0045] The data mapping business logic includes: instructing the operation unit to obtain a first target log from a file segment according to a first preset string, and generating a first key-value pair according to the first target log, using the first target log as the key and a preset value as the value in the first key-value pair; or instructing the operation unit to obtain a second target log from a file segment according to a second preset string, and obtaining a preset indicator value according to the second target log, using the preset indicator as the key and the value as the value in the second key-value pair.

[0046] Based on the mapping business logic, context file, and preset technical framework, generate a mapping kernel function;

[0047] Obtain grouping logic, wherein the grouping logic includes: placing the first key-value pairs with the same key into the same key-value pair group, or placing the second key-value pairs with the same key into the same key-value pair group;

[0048] Generate grouping functions based on grouping logic, context files, and a pre-defined technical framework;

[0049] The data merging business logic is obtained, which includes: instructing the operation unit to merge the value of the first key-value pair or the value of the second key-value pair in the key-value pair group;

[0050] Based on the data merging business logic, context files, and preset technical framework, a merging kernel function is generated.

[0051] In this embodiment, a mapping kernel function is generated based on the mapping business logic, context file, and preset technical framework; a grouping function is generated based on the grouping logic, context file, and preset technical framework; and a merging kernel function is generated based on the data merging business logic, context file, and preset technical framework. This enables the present invention to control the central processing unit's computing units to achieve parallel computing based on the mapping kernel function, grouping function, and merging kernel function.

[0052] Secondly, the present invention provides an apparatus for implementing parallel computing based on a central processing unit, the apparatus comprising:

[0053] The first acquisition module is used to acquire a context file and a file to be processed, and to slice the file to be processed according to the context file to obtain a first preset number of file slices. The context file is used to determine the operation unit, and the operation unit is contained in the central processing unit.

[0054] The second acquisition module is used to acquire the mapping kernel function, the grouping function, and the merging kernel function, wherein the mapping kernel function, the grouping function, and the merging kernel function are generated based on a preset technical framework;

[0055] The parallel computing module is used to generate a first preset number of mapping tasks based on file fragmentation and mapping kernel function, and to use the computing unit to perform parallel computing on the mapping tasks to obtain the processing results.

[0056] The grouping module is used to group the processing results using grouping functions to obtain the grouped results.

[0057] The merging module is used to generate a second preset number of merging tasks based on the merging kernel function and the grouping results. The processing unit is used to process the merging tasks to obtain the parallel computing results of the files to be processed.

[0058] Thirdly, the present invention provides a computer device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the method for parallel computing based on a central processing unit as described in the first aspect or any corresponding embodiment thereof.

[0059] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to execute the method for parallel computing based on a central processing unit according to the first aspect or any corresponding embodiment described above. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the specific embodiments or related technologies of the present invention, the drawings used in the description of the specific embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0061] Figure 1 This is a flowchart illustrating a method for implementing parallel computing based on a central processing unit according to an embodiment of the present invention;

[0062] Figure 2 This is a schematic diagram of the multi-CPU parallel computing process according to an embodiment of the present invention;

[0063] Figure 3 This is a structural block diagram of a device for implementing parallel computing based on a central processing unit according to an embodiment of the present invention;

[0064] Figure 4 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] Mainstream AI algorithm model training and inference environments rely on general-purpose GPUs and CUDA (Compute Unified Device Architecture) technology to achieve parallel computing. On the other hand, in the field of parallel computing, large-scale CPU clusters built on the Hadoop system are often used to meet computational requirements. However, Hadoop's CPU adaptability is limited and cannot meet general requirements. Current solutions for parallel computing, using technologies such as OpenCL and Hadoop, cannot achieve parallel computing solely using CPUs. Reasons include: OpenCL technology, simply by changing serial processing logic to parallelizable kernel functions, offers limited improvement in computing power; Hadoop technology has limited CPU support and cannot meet general requirements; OpenCL kernel functions cannot be directly converted to the MapReduce programming model, a programming model within Hadoop used for parallel computation of large-scale datasets. In summary, finding a general high-performance computing (HPC) method that fully utilizes mature multi-core CPU technology to improve the overall computing power of parallel computing and better complete AI training, inference, and big data processing tasks under limited resources and constraints is a pressing issue that needs to be addressed.

[0067] Based on the above, this invention provides a method for implementing parallel computing based on a central processing unit (CPU). It utilizes OpenCL (Open Computing Language) technology to achieve gridded management of computing units in a multi-CPU (multi-core) environment, simulating GPU-like computing units and structures to enhance parallel computing capabilities. OpenCL is a standard and framework for general-purpose parallel programming in heterogeneous systems. Based on the MapReduce model of Hadoop technology, Map and Reduce kernel functions are written to perform step-by-step CPU gridded parallel computing of data partitioning, loading, mapping, grouping, and merging processes, improving the efficiency of parallel computing. MapReduce is a programming model that can be used for large-scale data processing and analysis, including text analysis, log analysis, and machine learning. This achieves the goal of implementing parallel computing solely using the CPU, fully utilizing the limited computing resources of the CPU, and maximizing computing power without GPU assistance to meet the needs of AI algorithms and HPC.

[0068] According to an embodiment of the present invention, a method embodiment for parallel computing based on a central processing unit is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer device with data processing capabilities, such as a computer, a server, etc. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0069] This embodiment provides a method for implementing parallel computing based on a central processing unit, which can be used in the aforementioned computer equipment. Figure 1 This is a flowchart of a method for implementing parallel computing based on a central processing unit according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps:

[0070] Step S101: Obtain the context file and the file to be processed, and divide the file to be processed into fragments according to the context file to obtain a first preset number of file fragments. The context file is used to determine the operation unit, and the operation unit is contained in the central processing unit.

[0071] Specifically, the process involves obtaining a context file and files to be processed for parallel computation. These files can be, for example, files containing logs to be analyzed, files containing text to be analyzed, or files containing data to be input into a machine learning model for prediction. The context file defines the runtime environment of the entire pre-defined technical framework, OpenCL. The pre-defined technical framework includes OpenCL, a kernel, a device, program objects, and memory objects. The device is the computing device called by the OpenCL program; the kernel is the computing unit executed on the computer device. In this invention, the computing unit is obtained by meshing the central processing unit (CPU), with each CPU core serving as a computing unit; the program objects are the source code and executable file of the kernel program; and the memory objects are the variables required by the computer device to execute the OpenCL program.

[0072] The number of computation units can be obtained from the context file. The file to be processed is then divided into fragments based on the number of computation units, resulting in a first preset number of file fragments. The first preset number is the number of computation units, ensuring that each computation unit only needs to process one file fragment.

[0073] Step S102: Obtain the mapping kernel function, grouping function, and merging kernel function, wherein the mapping kernel function, grouping function, and merging kernel function are generated based on a preset technical framework.

[0074] Specifically, this involves obtaining the Map kernel, Shuffle kernel, and Reduce kernel. The pre-defined technical framework is OpenCL, and the Map, Shuffle, and Reduce kernels are all written in OpenCL. The Map kernel generates Map tasks, which instruct the computation units to process file slices. The Shuffle kernel groups the processing results from the computation units. The Reduce kernel generates Reduce tasks, which instruct the computation units to merge the grouped results.

[0075] Step S103: Generate a first preset number of mapping tasks based on file fragmentation and mapping kernel function, and use the computing unit to perform parallel computation on the mapping tasks to obtain the processing results.

[0076] Specifically, the Map kernel function is called to generate a first preset number of mapping tasks (Map tasks) based on the file fragments, so that each processing unit only needs to process one Map task. The Map kernel function contains data mapping business logic, such as: if the file fragments record data such as code or strings, then the number of each word appearing in the data of the file fragments is counted.

[0077] Each Map task is computed in parallel using a computing unit. The file fragments in the Map task are processed according to the data mapping business logic in the Map task to obtain the processing result of each computing unit. The processing result is represented in the form of key-value pairs.

[0078] Step S104: Use a grouping function to group the processing results to obtain the grouping results.

[0079] Specifically, the processing results are represented in the form of key-value pairs. When grouping the processing results using a grouping function, key-value pairs with the same key are grouped together to obtain the grouped results.

[0080] Step S105: Based on the merging kernel function and the grouping results, generate a second preset number of merging tasks, and use the computing unit to process the merging tasks to obtain the parallel computing results of the files to be processed.

[0081] Specifically, the Reduce kernel function is called to generate a second preset number of merge tasks (Reduce tasks) based on the grouping results. The second preset number represents multiple tasks. Each processing unit only needs to process one Reduce task at a time. The Reduce kernel function contains data merging business logic, such as merging the values ​​of key-value pairs in the same group in the grouping results.

[0082] The computing unit processes the Reduce task, merges different groups in the grouping result according to the data merging business logic in the Reduce task, and outputs the parallel computing result of the file to be processed. The parallel computing result is also represented by key-value pairs.

[0083] This embodiment provides a method for parallel computing based on a central processing unit (CPU). The file to be processed is fragmented. Based on the mapping kernel function and the file fragments, mapping tasks are generated and delivered to the CPU's processing unit for parallel computation. The processing results are grouped, and finally, based on the merging kernel function and the grouping results, merging tasks are generated and delivered to the CPU's processing unit for merging, yielding the parallel computation result. This method utilizes only the CPU for parallel computing, making full use of the CPU's limited computing resources. Without GPU assistance, it maximizes computing power to meet the demands of parallel computing. It solves the problem in related technologies where parallel computing relies too heavily on the graphics processing unit (GPU), making it impossible to perform high-efficiency parallel computing solely through the CPU.

[0084] In some alternative implementations, the method further includes, before obtaining the context file and the file to be processed:

[0085] Obtain a third preset number of central processing units, parameter information of the central processing units, and platform information of the preset technical framework. The parameter information is used to determine the core of the central processing unit, and the platform information is the information required to implement the preset technical framework.

[0086] Based on platform information and parameter information, the central processing unit is meshed using a preset technical framework to obtain a fourth preset number of processing units, where each processing unit corresponds to a core of the central processing unit.

[0087] A context file is generated based on the fourth preset quantity, parameter information, and platform information.

[0088] Specifically, this embodiment uses a preset technical framework to perform mesh processing on multiple CPUs. This preset technical framework is, for example, the OpenCL framework. OpenCL is an open, royalty-free standard used for cross-platform parallel programming of various accelerators in supercomputers, cloud servers, personal computers, mobile devices, and embedded platforms. OpenCL is a framework for writing programs for heterogeneous platforms, consisting of a language for writing kernel functions and a set of application programming interfaces (APIs) for defining and controlling the platform. In this embodiment, the heterogeneous platform consists of a third preset number of CPUs, for example, 10.

[0089] Let's take the OpenCL technology framework as an example for illustration:

[0090] This function retrieves a third preset number of CPUs, their parameter information, and OpenCL platform information. The parameter information includes the number of cores per CPU. Platform information is required to implement the preset technical framework; different manufacturers implement OpenCL differently, and each manufacturer's implementation constitutes a platform. Processors from different manufacturers can only run on different platforms; in other words, a platform only runs processors from the manufacturer implementing that platform. When programming with OpenCL, the selected platform must first be determined. The platform ID can be retrieved using the function `clGetPlatFormIDs(cl_uint num_entries, cl_platform_id*platforms, cl_uint*num_platforms)`, where `cl_uint num_entries` is the number of platform IDs that `platforms` can store; `cl_platform_id*platforms` is a pointer used to store the found platforms; and `cl_uint*num_platforms` stores the number of platforms retrieved. A return value of 0 indicates program completion; a non-zero value indicates an error code.

[0091] The above process can be implemented using the following code:

[0092] ret = clGetPlatformIDs(); / / clGetPlatformIDs() is used to obtain the platform and perform initialization.

[0093] devs = clGetDeviceIDs(ret); / / clGetFileSplit is used to obtain device information and multi-CPU information.

[0094] Here, ret represents platform information; devs represents CPU parameter information.

[0095] Based on the platform information and parameters such as the number of CPUs and cores in OpenCL, the central processing unit is meshed using the OpenCL technology framework to obtain a fourth preset number of computing units. The fourth preset number = the number of CPUs * the number of CPU cores. The number of meshes determines the number of computing units that can perform parallel computing. That is, each mesh is a computing unit, and each computing unit corresponds to a core of the central processing unit. The OpenCL main program performs unified initialization and management of all computing units, simulating computing units and structures similar to GPUs, so as to significantly improve parallel processing capabilities.

[0096] The steps for calculating the number of grid cells are implemented using the following code, taking a server host with four CPUs, each containing 32 cores:

[0097] kels = 4 * 32 = 128; / / Calculate the number of grid cells, number of grid cells = number of CPUs * number of CPU cores

[0098] Here, kels is the number of grids. Since each grid is a computational unit, kels is also the number of computational units, i.e., the fourth preset number.

[0099] The following code generates a context file based on the fourth preset quantity, parameter information, and platform information:

[0100] context = clCreateContext(&ret,&devs,&kels); / / clCreateContext is used to create a context file, which is used to manage the mesh in a unified manner.

[0101] It's important to note that the OpenCL main program creates a context file, writing the number of compute units, parameter information, and CPU device parameter information into it. This context file indicates which devices will be used for subsequent kernel computations. Later, when the OpenCL main program calls the corresponding APIs to create kernel objects, program objects, and other objects, it passes the already created context as a parameter, indicating that these objects are all created within this passed context, thus establishing the corresponding relationships.

[0102] In this embodiment, the central processing unit is gridded using a preset technical framework to generate multiple computing units. These computing units can execute computing tasks simultaneously, and the parallel processing capability of the present invention is significantly improved through the use of multiple computing units.

[0103] In some optional implementations, the mapping task is computed in parallel using a computing unit to obtain the processing result, including:

[0104] The processing unit obtains the data mapping business logic and file fragmentation based on the mapping task;

[0105] The processing unit generates a fifth preset number of key-value pairs based on the data mapping business logic and file fragmentation;

[0106] The key-value pairs of all operation units are summarized to obtain the processing result.

[0107] Specifically, the Map kernel function is used to implement parallel computing in the data mapping stage. The Map kernel function generates Map tasks based on file fragments. The number of Map tasks is the same as the number of computing units. The OpenCL main program uniformly calls the computing units to perform parallel computing on the Map tasks and receives the computing results.

[0108] After the OpenCL main program calls the computation unit to perform parallel computation on the Map task, the computation unit obtains the data mapping business logic and file fragments of the Map kernel function from the Map task. The data mapping business logic is as follows: if the file fragment records data such as code and strings, then the number of each word appearing in the data of the file fragment is counted; if the file fragment records logs of accumulated information such as access time and access count, then the sum of each type of accumulated information in the log of the file fragment is counted; if the file fragment records logs of information such as access address and operation, then the number of each type of log appearing in the file fragment is counted.

[0109] Each processing unit processes the data in the file slices according to the data mapping business logic, uses key-value pairs to represent the processing results, and generates a fifth preset number of key-value pairs. The fifth number indicates multiple pairs. For example, if the data mapping business logic is to count the number of each word appearing in the data of the file slices, the processing unit reads the string in the file slices, uses the words appearing in the string as the keys of the key-value pairs, and uses the number of words as the values ​​of the key-value pairs, generating multiple key-value pairs.

[0110] The key-value pairs of all operation units are summarized to obtain the processing result.

[0111] The above process can be implemented using the following code:

[0112] mtk = clCreateKernel(map_task); / / Create the Map kernel function, mtk represents the Map kernel function

[0113] mapnd = clEnqueueNDRangeKernel(mtk); / / Execute the Map kernel function to generate the Map task (mapnd).

[0114] maps<key,value> =clEnqueueMapBuffer(&mapnd); / / Perform parallel computation on the Map task (mapnd) using the computation unit and return the processing result, which is represented by a map structure consisting of key-value pairs.

[0115] In this embodiment, the computing unit performs parallel computation on the mapping task to obtain the processing result, which improves the efficiency of the present invention in processing the file to be processed; the data mapping business logic is used to instruct the computing unit on how to calculate, which improves the accuracy and stability of the calculation.

[0116] In some optional implementations, the processing results are grouped using a grouping function to obtain grouped results, including:

[0117] The processing results yield key-value pairs, where each key-value pair contains a key;

[0118] Based on the key, the key-value pairs are grouped to obtain the grouping results. The grouping results contain a second preset number of key-value pair groups, and each key-value pair group contains a sixth preset number of key-value pairs with the same key.

[0119] The grouping results are passed to the merge kernel function.

[0120] Specifically, this invention utilizes key-value pairs to represent the processing results of the parallel computation of the mapping task by the arithmetic unit. The OpenCL main program calls the grouping function (Shuffle function), which is a component of the OpenCL main program.

[0121] The Shuffle function reads the processing results, obtaining key-value pairs. Each key-value pair contains a key and its corresponding value. The Shuffle function groups key-value pairs with the same key into the same key-value pair group. Each key-value pair group contains a sixth preset number of key-value pairs (this sixth preset number represents multiple groups, with no specific limit). The Shuffle function generates a second preset number of key-value pair groups, the same number as the subsequent merge tasks, because each key-value pair group needs to be processed by a merge task. These second preset number of key-value pair groups together constitute the grouping result of the Shuffle function. For example, if the data mapping business logic is to count the number of each word appearing in the data of a file shard, the processing unit reads the string from the file shard, using the words appearing in the string as the keys of key-value pairs and the number of words as the values. When the Shuffle function groups, it puts key-value pairs with the same value into the same key-value pair group. Therefore, each key-value pair in the same key-value pair group describes the same word, and the value of each key-value pair represents the number of times that word appears.

[0122] The Shuffle function groups the processing results and then passes the grouped results to the Reduce kernel function.

[0123] In this implementation, a grouping function is used to group key-value pairs with the same key into the same key-value pair group, which facilitates the subsequent merging of processing results, simplifies the merging business logic in the subsequent merging process, avoids merging errors, and prevents them from affecting the final parallel computing results.

[0124] In some optional implementations, a second preset number of merging tasks are generated based on the merging kernel function and the grouping results. The merging tasks are then processed using a computing unit to obtain the parallel computation results of the file to be processed, including:

[0125] Based on the grouping results, a second preset number of key-value pair groups are obtained;

[0126] Based on the merge kernel function and key-value pair grouping, a second preset number of merge tasks are generated, wherein the merge kernel function is used to write the data merging business logic into the merge task;

[0127] When processing the merging task, the processing unit merges the values ​​of key-value pairs in the key-value pair group according to the data merging business logic, and obtains the merging result of each key-value pair group.

[0128] The results of merging all key-value pairs are summarized to obtain the parallel computation results.

[0129] Specifically, after grouping the processing results, the Shuffle function passes the grouping results to the Reduce kernel function, which then obtains a second preset number of key-value pair groups from the grouping results.

[0130] The OpenCL main program calls the Reduce kernel function. The Reduce kernel function generates a second preset number of merge tasks (Reduce tasks) for each key-value pair group. Each Reduce task contains the data merging business logic of the Reduce kernel function, such as merging the values ​​of key-value pairs in the same group in the grouping results.

[0131] The OpenCL main program calls the computation unit to process the Reduce task. The computation unit, based on the data merging business logic, merges the values ​​of key-value pairs in the key-value pair groups, obtaining the merged result for each group. For example, if the data mapping business logic is to count the occurrences of each word in a file shard, the computation unit reads the string from the file shard, using the words as the keys and the word counts as the values. When the Shuffle function groups, key-value pairs with the same value are placed in the same group. Each key-value pair in the same group describes the same word, and the value represents the number of times that word appears. When the computation unit processes the Reduce task, it adds the values ​​of each key-value pair in the same group to obtain a target key-value pair. The key in this target key-value pair is the word, and the value is the total number of times that word appears in the entire file to be processed.

[0132] The merged results of all key-value pair groups are summarized to obtain the parallel computing results, which contain the same number of target key-value pairs as the key-value pair groups.

[0133] The above process can be implemented using the following code:

[0134] rtk = clCreateKernel(reduce_task); / / Creates the Reduce kernel function, rtk represents the Reduce kernel function

[0135] reducend = clEnqueueNDRangeKernel(rtk); / / Execute the Reduce kernel function to generate Reduce tasks (reducend).

[0136] Reduces<key,value> =clEnqueueMapBuffer(&reducend); / / Merge the Reduce task (reducend) using the computation unit and return the parallel computation result reduces, which is represented by key-value pairs.

[0137] In this implementation, generating merge tasks assigns each merge task to a processing unit, improving merge efficiency. Utilizing data merging business logic instructs the processing unit on how to merge, enhancing the accuracy and stability of the merge process.

[0138] In some optional implementations, the file to be processed is fragmented according to the context file to obtain a first preset number of file fragments, including:

[0139] The server resource threshold, the fourth preset number of computing units, and the memory capacity threshold are obtained from the context file.

[0140] Get the file size of the file to be processed;

[0141] If the file size is equal to or less than the memory capacity threshold, the file fragment size is obtained based on the fourth preset quantity, the file size, and the server resource threshold.

[0142] Based on the file fragment size and the file to be processed, a first preset number of file fragments are obtained.

[0143] Specifically, the file to be processed is divided into file slices according to the number of processing units, resulting in file slices with the same number of processing units. This ensures that each subsequent processing unit only needs to process one file slice. The specific steps include:

[0144] The server resource thresholds, the fourth preset number of computing units, and the server memory capacity threshold are obtained from the context file. For example, if a server host has four CPUs, each with 32 cores, and 128GB of memory, then the fourth preset number is 128, and the memory capacity threshold is 128GB. Server resources include hardware and software resources. Hardware resources include processors, memory, and storage devices, which can improve the overall system's computing power and storage capacity. Software resources include the operating system, database management system, and applications; other computers can access the software resources on the server via the network. Therefore, server resource thresholds include, for example, the upper limit of computing power and the upper limit of storage capacity.

[0145] If the file size of the file to be processed is equal to or less than (but not greater than) the aforementioned memory capacity threshold, the file size is divided by the fourth preset number to obtain the file fragment size. It is then determined whether dividing the file to be processed according to the file fragment size would exceed the server resource threshold. For example, is the file fragment size too large, exceeding the computing power limit of the processing unit and causing excessive processing time? Is the file fragment size too large, exceeding the storage capacity limit corresponding to the processing unit? If the file fragment size does not exceed the server resource threshold, the file to be processed is divided according to the file fragment size. For example, if the file size is 128G and the fourth preset number is 128, the file fragment size is 1G. If 1G does not exceed the server resource threshold, the file to be processed is divided into multiple file fragments of 1G each. Based on the file fragment size and the file to be processed, a first preset number of file fragments are obtained, the same number as the number of subsequent Map tasks. The starting position pointers (urls) of the grouped files are then handed over to the OpenCL main program for unified loading.

[0146] The above process can be implemented using the following code:

[0147] fls = clGetFileSplit(&url,&kels,&context); / / clGetFileSplit is a regular function for splitting files. It is used to divide the file to be processed into multiple file fragments based on the context file. The size of the file fragment is the file size of the file to be processed / the number of operation units (kels), and records the file start position pointer (url) of the file fragment.

[0148] In this embodiment, the file fragment size is first determined, and the file to be processed is fragmented according to the file fragment size to obtain a first preset number of file fragments. This allows different processing units to process the file to be processed simultaneously, and only one file fragment needs to be processed, thus improving the computational efficiency of parallel computing.

[0149] In some optional implementations, after obtaining the parallel computation results of the file to be processed, the method further includes:

[0150] Obtain the actual business requirements, and obtain the business logic implementation functions based on the actual business requirements;

[0151] Based on the business logic implementation function, write the parallel computing results into the business logic implementation path and / or output the parallel computing results.

[0152] Specifically, it involves obtaining actual business needs, such as: printing out the parallel computing results; writing the parallel computing results into a target file; and storing the parallel computing results in a database.

[0153] Obtain the business logic implementation function according to the actual business needs. For example, if the actual business needs is to print out the parallel computing results, then obtain the function that can print out the parallel computing results as the business logic implementation function; if the actual business needs is to write the parallel computing results to the target file, then obtain the function that can write the parallel computing results to the target path as the business logic implementation function; the actual business needs can include both of the above situations at the same time.

[0154] If the actual business requirement is to print out the parallel computing results, then calling the business logic implementation function will output the parallel computing results, i.e., print the parallel computing results. If the actual business requirement is to write the parallel computing results to a target file, then calling the business logic implementation function will write the parallel computing results to the business logic implementation path, i.e., write to the target file. If the actual business requirement includes both of the above, then calling the business logic implementation function will write the parallel computing results to the target file under the business logic implementation path and print the parallel computing results.

[0155] In this embodiment, the business logic implementation function is obtained according to actual business needs, executed, and the parallel computing results are written into the business logic implementation path and / or output. This enables the invention to have more functions and expands its scope of application.

[0156] In some alternative implementations, the method further includes, before obtaining the mapping kernel function, the grouping function, and the merging kernel function:

[0157] The data mapping business logic includes: instructing the operation unit to obtain a first target log from a file segment according to a first preset string, and generating a first key-value pair according to the first target log, using the first target log as the key and a preset value as the value in the first key-value pair; or instructing the operation unit to obtain a second target log from a file segment according to a second preset string, and obtaining a preset indicator value according to the second target log, using the preset indicator as the key and the value as the value in the second key-value pair.

[0158] Based on the mapping business logic, context file, and preset technical framework, generate a mapping kernel function;

[0159] Obtain grouping logic, wherein the grouping logic includes: placing the first key-value pairs with the same key into the same key-value pair group, or placing the second key-value pairs with the same key into the same key-value pair group;

[0160] Generate grouping functions based on grouping logic, context files, and a pre-defined technical framework;

[0161] The data merging business logic is obtained, which includes: instructing the operation unit to merge the value of the first key-value pair or the value of the second key-value pair in the key-value pair group;

[0162] Based on the data merging business logic, context files, and preset technical framework, a merging kernel function is generated.

[0163] Specifically, this invention utilizes the MapReduce parallel programming model from Hadoop technology to provide parallel computing for massive amounts of data. Including mapping (Map) operations and reducing (Reduce) operations, this design undoubtedly greatly reduces the difficulty of developing distributed parallel programs.

[0164] Write a Map kernel function to implement parallel computation during the data mapping phase. First, obtain the data mapping business logic in the Map kernel function. The data mapping business logic is as follows: 1. If the file shard records logs containing information such as visitor ID, visitor user information, and request time, then count the number of occurrences of each type of log in the file shard. Specifically, this includes: instructing the computation unit to obtain the first target log from the file shard based on a first preset string. The first preset string is, for example, remote_addrstring (representing visitor ID), remote_addr string (representing visitor user information), time_local string (representing request time), etc. The first target log filtered based on the first preset string is the log containing information such as visitor ID, visitor user information, and request time; generating the first key-value pair based on the first target log, using the first target log as the key and a preset value as the value. The preset value is, for example, 1, indicating that the first target log appeared once. 2. If the file shard records logs containing cumulative information such as page dwell time (seconds) and session page visits, then the sum of each type of cumulative information in the file shard's log is calculated. Specifically, this includes: instructing the processing unit to obtain the second target log from the file shard based on the second preset string. The second preset string is, for example, page_staylong string (page dwell time), pageVisits int (session page visits), etc. The second target log filtered based on the second preset string is the log containing cumulative information such as page dwell time (seconds) and session page visits; obtaining the value of a preset indicator based on the second target log. The preset indicator is, for example, page dwell time (seconds) and session page visits. Correspondingly, the data of the preset indicator is, for example, page dwell time of 30 seconds and session page visits of 100 times; using the preset indicator as the key in the second key-value pair, and using the value corresponding to the preset indicator as the value in the second key-value pair.

[0165] Based on the above data mapping business logic, context file, and preset technical framework, a mapping kernel function (Map kernel function) is generated. The preset technical framework, for example, is the OpenCL technical framework, which can be implemented using the following code:

[0166] __kernel void map_task(*fls,&context){ / / Write the Map kernel function using the context file

[0167] MapLG / / Write the data mapping business logic (MapLG)

[0168] maps<key1,value1> =List(fls); / / Generates a key-value pair mapping, List(fls) represents file fragments, maps<key1,value1> The processing results are represented by a map structure consisting of key-value pairs.

[0169] }

[0170] Write a grouping function (Shuffle function). The Shuffle function groups the processing results returned by the Map kernel function and passes the grouped results to the Reduce kernel function. The Shuffle function is a component of the OpenCL main program. The writing process includes: obtaining the grouping logic, such as grouping first key-value pairs with the same key into the same key-value pair group, or grouping second key-value pairs with the same key into the same key-value pair group. Based on the grouping logic, the context file, and the preset technology framework, generate the grouping function (Shuffle function). The preset technology framework is, for example, the OpenCL technology framework. This can be implemented using the following code:

[0171] void Shuffle_task(&maps,&context){ / / Write the Shuffle function based on the context file shuffles[<key2,value2> = Range(maps, context); / / Grouping logic: put key-value pairs with the same key into the same key-value pair group shuffles. The range to be grouped is determined by the processing results maps and context files returned by the Map kernel function Range(maps, context)

[0172] }

[0173] Write a Reduce kernel function to implement parallel computation in the data merging phase. This generates Reduce tasks for processing by the computation unit. The Reduce kernel function is uniformly invoked by OpenCL, and the parallel computation results returned by the Reduce kernel function are received. The writing process includes: obtaining the data merging business logic, such as: if the key-value pair group contains the first key-value pair, instruct the computation unit to merge the values ​​of the first key-value pair; if the key-value pair group contains the second key-value pair, instruct the computation unit to merge the values ​​of the second key-value pair. Based on the data merging business logic, the context file, and the preset technical framework, generate the Reduce kernel function. The preset technical framework is, for example, the OpenCL technical framework, which can be implemented using the following code:

[0174] __kernel void Reduce_task(&shuffles,&context){ / / Write the Reduce kernel function

[0175] ReduceLG / / Write the data merging business logic (ReduceLG)

[0176] Reduces<key3,value3> =Count(shuffles); / / Performs a merge count (Count) on the key-value pairs in the grouped results and returns the parallel computation result (reduces).<key3,value3>

[0177] }

[0178] In this embodiment, a mapping kernel function is generated based on the mapping business logic, context file, and preset technical framework; a grouping function is generated based on the grouping logic, context file, and preset technical framework; and a merging kernel function is generated based on the data merging business logic, context file, and preset technical framework. This enables the present invention to control the central processing unit's computing units to achieve parallel computing based on the mapping kernel function, grouping function, and merging kernel function.

[0179] This embodiment provides a method for implementing parallel computing based on a central processing unit, which can be used in the aforementioned computer devices, such as... Figure 2 As shown, the process includes the following steps:

[0180] Mesh processing of multiple CPUs (multiple CPUs, multiple cores) is performed using the OpenCI technology framework;

[0181] Using the OpenCL main program, the platform is obtained and initialized; the device is obtained to get multi-CPU information; the number of grids is calculated; and a context is created for unified management of the grids.

[0182] The data is grouped according to the number of grid cells and then loaded uniformly by the OpenCL main program;

[0183] Write a Map kernel function, which is called uniformly by the OpenCL main program and receives the calculation results;

[0184] Write a grouping function called Shuffle to group the results returned by the Map kernel function and pass the results to the Reduce kernel function;

[0185] Write a Reduce kernel function to implement parallel computing during the data merging phase;

[0186] Based on the calculation results and actual business needs, design and execute the business processing procedures.

[0187] In this embodiment, by utilizing only the CPU, limited resources are fully utilized to maximize computing power without the constraints of GPU assistance, thereby meeting the parallel computing needs of business applications for AI training and inference and big data processing.

[0188] This embodiment also provides a device for parallel computing based on a central processing unit. This device is used to implement the above embodiments and preferred embodiments, and details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0189] This embodiment provides a device for implementing parallel computing based on a central processing unit, such as... Figure 3 As shown, it includes:

[0190] The first acquisition module 301 is used to acquire a context file and a file to be processed, and to slice the file to be processed according to the context file to obtain a first preset number of file slices, wherein the context file is used to determine the operation unit, and the operation unit is contained in the central processing unit.

[0191] The second acquisition module 302 is used to acquire the mapping kernel function, the grouping function, and the merging kernel function, wherein the mapping kernel function, the grouping function, and the merging kernel function are generated based on a preset technical framework;

[0192] Parallel computing module 303 is used to generate a first preset number of mapping tasks based on file fragmentation and mapping kernel function, and to use the computing unit to perform parallel computing on the mapping tasks to obtain the processing results.

[0193] Grouping module 304 is used to group the processing results using a grouping function to obtain the grouped results;

[0194] The merging module 305 is used to generate a second preset number of merging tasks based on the merging kernel function and the grouping results, and to process the merging tasks using the computing unit to obtain the parallel computing results of the files to be processed.

[0195] In some alternative embodiments, the device further includes:

[0196] The third acquisition module is used to acquire a third preset number of central processing units, parameter information of the central processing units, and platform information of the preset technical framework. The parameter information is used to determine the core of the central processing unit, and the platform information is the information required to implement the preset technical framework.

[0197] The gridding processing module is used to perform gridding processing on the central processing unit based on platform information and parameter information using a preset technical framework to obtain a fourth preset number of processing units, wherein each processing unit corresponds to a core of the central processing unit;

[0198] The first generation module is used to generate a context file based on the fourth preset quantity, parameter information, and platform information.

[0199] In some alternative implementations, the parallel computing module 303 includes:

[0200] The first obtaining unit is used by the computing unit to obtain data mapping business logic and file fragments based on the mapping task;

[0201] The first generation unit is used by the computing unit to generate a fifth preset number of key-value pairs based on the data mapping business logic and file fragmentation.

[0202] The first summarization unit is used to summarize the key-value pairs of all operation units to obtain the processing result.

[0203] In some alternative implementations, the grouping module 304 includes:

[0204] The second obtaining unit is used to obtain key-value pairs based on the processing result, wherein each key-value pair contains a key;

[0205] The third obtaining unit is used to group key-value pairs according to the key to obtain a grouping result, wherein the grouping result contains a second preset number of key-value pair groups, and each key-value pair group contains a sixth preset number of key-value pairs with the same key;

[0206] The passing unit is used to pass the grouping results to the merge kernel function.

[0207] In some alternative implementations, the merging module 305 includes:

[0208] The fourth obtaining unit is used to obtain a second preset number of key-value pair groups based on the grouping results;

[0209] The second generation unit is used to generate a second preset number of merging tasks based on the merging kernel function and key-value pair grouping, wherein the merging kernel function is used to write the data merging business logic into the merging tasks;

[0210] The merging unit is used by the processing unit to merge the values ​​of key-value pairs in the key-value pair group according to the data merging business logic when processing merging tasks, so as to obtain the merging result of each key-value pair group.

[0211] The second summarization unit is used to summarize the merge results of all key-value pair groups to obtain the parallel computing results.

[0212] In some optional implementations, the first acquisition module 301 includes:

[0213] The fifth obtaining unit is used to obtain the server resource threshold, the fourth preset number of computing units, and the memory capacity threshold based on the context file;

[0214] The acquisition unit is used to obtain the file size of the file to be processed.

[0215] The sixth obtaining unit is used to obtain the file fragment size based on the fourth preset quantity, the file size, and the server resource threshold when the file size is equal to or less than the memory capacity threshold;

[0216] The seventh obtaining unit is used to obtain a first preset number of file fragments based on the file fragment size and the file to be processed.

[0217] In some alternative embodiments, the device further includes:

[0218] The fourth acquisition module is used to acquire the actual business needs and acquire the business logic implementation functions based on the actual business needs;

[0219] The execution module is used to implement functions according to business logic, write the parallel computing results into the business logic implementation path and / or output the parallel computing results.

[0220] In some alternative embodiments, the device further includes:

[0221] The fifth acquisition module is used to acquire data mapping business logic, wherein the data mapping business logic includes: instructing the operation unit to acquire a first target log from the file segment according to a first preset string, and generate a first key-value pair according to the first target log, using the first target log as the key in the first key-value pair and a preset value as the value in the first key-value pair; or instructing the operation unit to acquire a second target log from the file segment according to a second preset string, and obtain the value of a preset indicator according to the second target log, using the preset indicator as the key in the second key-value pair and the value as the value in the second key-value pair.

[0222] The second generation module is used to generate a mapping kernel function based on the mapping business logic, context file, and preset technical framework;

[0223] The sixth acquisition module is used to acquire grouping logic, wherein the grouping logic includes: putting first key-value pairs with the same key into the same key-value pair group, or putting second key-value pairs with the same key into the same key-value pair group;

[0224] The third generation module is used to generate grouping functions based on grouping logic, context files, and preset technical frameworks;

[0225] The seventh acquisition module is used to acquire data merging business logic, wherein the data merging business logic includes: instructing the operation unit to merge the value of the first key-value pair or the value of the second key-value pair in the key-value pair group;

[0226] The fourth generation module is used to generate a merging kernel function based on the data merging business logic, context files, and a preset technical framework.

[0227] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0228] In this embodiment, the device for parallel computing based on a central processing unit is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0229] This invention also provides a computer device having the above-described features. Figure 3 The device shown is a parallel computing device based on a central processing unit.

[0230] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 4 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 4 Take a processor 10 as an example.

[0231] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0232] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0233] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0234] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0235] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.

[0236] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0237] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and all such modifications and variations fall within the scope defined by the appended claims.

Claims

1. A method for implementing parallel computing based on a central processing unit, characterized in that, The method includes: Obtain a third preset number of central processing units (CPUs), parameter information of the CPUs, and platform information of a preset technical framework, wherein the parameter information is used to determine the core of the CPUs, and the platform information is the information required to implement the preset technical framework, wherein the preset technical framework is OpenCL; Based on the platform information and the parameter information, the central processing unit is subjected to grid-based processing using a preset technical framework to obtain a fourth preset number of processing units, wherein each processing unit corresponds to one core of the central processing unit; A context file is generated based on the fourth preset quantity, the parameter information, and the platform information; Obtain a context file and a file to be processed, and segment the file to be processed according to the context file to obtain a first preset number of file segments, wherein the context file is used to determine the operation unit, and the operation unit is contained in the central processing unit; Obtain the mapping kernel function, the grouping function, and the merging kernel function, wherein the mapping kernel function, the grouping function, and the merging kernel function are generated based on a preset technical framework; The first preset number of mapping tasks are generated based on the file fragments and the mapping kernel function. The mapping tasks are then computed in parallel using the computing unit to obtain the processing results. The processing results are grouped using the grouping function to obtain grouping results; Based on the merging kernel function and the grouping results, a second preset number of merging tasks are generated. The merging tasks are then processed using the computing unit to obtain the parallel computing results of the file to be processed.

2. The method according to claim 1, characterized in that, The parallel computation of the mapping task using the computing unit to obtain the processing result includes: The processing unit obtains the data mapping business logic and the file fragmentation based on the mapping task; The processing unit generates a fifth preset number of key-value pairs based on the data mapping business logic and the file fragmentation; The key-value pairs of all the aforementioned computational units are aggregated to obtain the processing result.

3. The method according to claim 2, characterized in that, The step of grouping the processing results using the grouping function to obtain grouped results includes: The key-value pairs are obtained based on the processing result, wherein each key-value pair contains a key; Based on the key, the key-value pairs are grouped to obtain the grouping result, wherein the grouping result contains a second preset number of key-value pair groups, and each key-value pair group contains a sixth preset number of key-value pairs with the same key; The grouping results are passed to the merge kernel function.

4. The method according to claim 3, characterized in that, The step of generating a second preset number of merging tasks based on the merging kernel function and the grouping results, and processing the merging tasks using the computing unit to obtain the parallel computing results of the file to be processed includes: Based on the grouping results, a second preset number of key-value pair groups are obtained; Based on the merging kernel function and the key-value pair grouping, a second preset number of merging tasks are generated, wherein the merging kernel function is used to write the data merging business logic into the merging task; When processing the merging task, the computing unit merges the values ​​of the key-value pairs in the key-value pair group according to the data merging business logic to obtain the merging result of each key-value pair group. The merging results of all the key-value pair groups are summarized to obtain the parallel computing result.

5. The method according to claim 1, characterized in that, The step of segmenting the file to be processed according to the context file to obtain a first preset number of file segments includes: The server resource threshold, the fourth preset number of computing units, and the memory capacity threshold are obtained based on the context file. Obtain the file size of the file to be processed; If the file size is equal to or less than the memory capacity threshold, the file fragment size is obtained based on the fourth preset quantity, the file size, and the server resource threshold. Based on the file fragment size and the file to be processed, a first preset number of file fragments are obtained.

6. The method according to claim 1, characterized in that, After obtaining the parallel computation results of the file to be processed, the method further includes: Obtain the actual business needs, and obtain the business logic implementation functions based on the actual business needs; According to the business logic implementation function, the parallel computing result is written into the business logic implementation path and / or the parallel computing result is output.

7. The method according to claim 1, characterized in that, Before obtaining the mapping kernel function, grouping function, and merging kernel function, the method further includes: The data mapping business logic includes: instructing the operation unit to obtain a first target log from the file segment according to a first preset string, and generate a first key-value pair according to the first target log, using the first target log as the key and a preset value as the value of the first key-value pair; or, instructing the operation unit to obtain a second target log from the file segment according to a second preset string, and obtain a value of a preset indicator according to the second target log, using the preset indicator as the key and the value as the value of the second key-value pair. The mapping kernel function is generated based on the data mapping business logic, the context file, and the preset technical framework; Obtain grouping logic, wherein the grouping logic includes: placing first key-value pairs with the same key into the same key-value pair group, or placing second key-value pairs with the same key into the same key-value pair group; The grouping function is generated based on the grouping logic, the context file, and the preset technical framework; The data merging business logic includes: instructing the operation unit to merge the values ​​of the first key-value pair or the second key-value pair in the key-value pair group; The merging kernel function is generated based on the data merging business logic, the context file, and the preset technical framework.

8. A device for parallel computing based on a central processing unit, characterized in that, The device includes: The third acquisition module is used to acquire a third preset number of central processing units, parameter information of the central processing units, and platform information of a preset technical framework. The parameter information is used to determine the core of the central processing unit, and the platform information is the information required to implement the preset technical framework, wherein the preset technical framework is OpenCL. The gridding processing module is used to perform gridding processing on the central processing unit based on the platform information and the parameter information using a preset technical framework to obtain a fourth preset number of processing units, wherein each processing unit corresponds to one core of the central processing unit; The first generation module is used to generate a context file based on the fourth preset quantity, the parameter information, and the platform information; The first acquisition module is used to acquire a context file and a file to be processed, and to segment the file to be processed according to the context file to obtain a first preset number of file segments, wherein the context file is used to determine the operation unit, and the operation unit is contained in the central processing unit. The second acquisition module is used to acquire the mapping kernel function, the grouping function, and the merging kernel function, wherein the mapping kernel function, the grouping function, and the merging kernel function are generated based on a preset technical framework; The parallel computing module is used to generate a first preset number of mapping tasks based on the file fragments and the mapping kernel function, and to perform parallel computing on the mapping tasks using the computing unit to obtain the processing results. The grouping module is used to group the processing results using the grouping function to obtain grouping results; The merging module is used to generate a second preset number of merging tasks based on the merging kernel function and the grouping results, and to process the merging tasks using the computing unit to obtain the parallel computing results of the file to be processed.

9. A computer device, characterized in that, include: A memory and a processor are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to perform the method of parallel computing based on a central processing unit as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the method of parallel computing based on a central processing unit as described in any one of claims 1 to 7.

Images