A distributed grid processing method, device and equipment of data and medium

Abstract

Embodiments of the present application disclose a kind of data distributed grid processing method, device, equipment and medium. Among them, the method comprises: the data to be processed is split into atomic operation queue and parameter queue;The time overhead of each atomic operation in atomic operation queue is calculated by time trial calculator;According to the time overhead of each atomic operation, and the processing capacity of distributed grid computing node, each atomic operation in atomic operation queue is distributed based on dependency relationship;The processing result of distributed grid computing node is obtained, and the distributed grid processing result is obtained based on the merging of dependency relationship.This technical solution, by splitting the data to be processed, can adapt to the acceleration scheme of grid technology, by calculating the time overhead of atomic operation and the processing capacity of distributed grid computing node, atomic operation can be reasonably distributed, to achieve the effect of greatly improving system real-time.

Description

Technical Field

[0001] This invention relates to the field of Internet technology, and in particular to a distributed grid-based data processing method, apparatus, device, and medium. Background Technology

[0002] With the development of mobile internet, some data processing tasks in transaction management systems have high real-time requirements. How to accelerate complex business calculations and improve the real-time performance of business processing is a pressing issue that transaction management systems need to address.

[0003] The main solutions currently available are to accelerate complex computation requests through multithreading or increasing the number of server nodes. Multithreading requires that the logic for processing tasks cannot depend on intermediate results, thus enabling parallel decomposition; increasing the number of server nodes is mainly used to accelerate batch atomic operation requests.

[0004] However, existing complex logic processing tasks often involve hierarchical or sequential structures where each step depends on the previous one, making them unsuitable for multi-threaded acceleration. Therefore, multi-threading does not effectively improve the real-time performance of the entire processing flow. Increasing the number of server nodes is also insufficient for tasks with a small total number of requests and time-consuming individual atomic operations. Summary of the Invention

[0005] This invention provides a distributed gridded data processing method, apparatus, device, and medium. By splitting the data to be processed, it can adapt to the acceleration scheme of gridded technology. By calculating the time overhead of atomic operations and the processing capabilities of distributed gridded computing nodes, it can rationally allocate atomic operations and achieve a significant improvement in the real-time performance of the system.

[0006] According to one aspect of the present invention, a distributed gridded data processing method is provided, the method comprising:

[0007] The processing logic of the data to be processed is obtained through a preset rule engine, and the data to be processed is split into an atomic operation queue and a parameter queue; wherein, the atomic operation queue contains the dependencies between the atomic operations;

[0008] The time cost of each atomic operation in the atomic operation queue is calculated using a time calculator.

[0009] Based on the time cost of each atomic operation and the processing capacity of the distributed grid computing nodes, each atomic operation in the atomic operation queue is distributed according to the dependency relationship.

[0010] Obtain the processing results of the distributed grid computing nodes, and merge them based on the dependencies to obtain the distributed grid processing results.

[0011] According to another aspect of the present invention, a distributed gridded data processing apparatus is provided, comprising:

[0012] The pending data splitting module is used to obtain the processing logic of the pending data through a preset rule engine, and split the pending data into an atomic operation queue and a parameter queue; wherein, the atomic operation queue contains the dependencies between each atomic operation;

[0013] The time cost calculation module is used to calculate the time cost of each atomic operation in the atomic operation queue using a time calculator.

[0014] An atomic operation distribution module is used to distribute each atomic operation in the atomic operation queue based on the dependency relationship, according to the time cost of each atomic operation and the processing capacity of the distributed grid computing nodes.

[0015] The processing result acquisition module is used to acquire the processing results of the distributed grid computing nodes and merge them based on the dependency relationship to obtain the distributed grid processing result.

[0016] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0017] At least one processor; and a memory communicatively connected to said at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the distributed gridded data processing method according to any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the distributed gridded data processing method described in any embodiment of the present invention.

[0020] The technical solution of this invention, by splitting the data to be processed and accelerating the process to adapt to grid technology, and by utilizing the time overhead of atomic operations and the processing capabilities of distributed grid computing nodes, can reasonably allocate atomic operations and achieve a significant improvement in the real-time performance of the system.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of a distributed gridded data processing method provided in Embodiment 1 of the present invention;

[0024] Figure 2 This is a flowchart of a distributed gridded data processing method provided in Embodiment 2 of the present invention;

[0025] Figure 3 This is a technical solution system diagram provided in Embodiment 3 of the present invention;

[0026] Figure 4 This is a network diagram of bond price yield in the bond full price calculation scenario provided by Embodiment 3 of the present invention;

[0027] Figure 5 This is a schematic diagram of the structure of a distributed gridded data processing device according to Embodiment 4 of the present invention;

[0028] Figure 6 This is a schematic diagram of the structure of an electronic device provided according to Embodiment Six of the present invention. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first," "second," "target," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] Example 1

[0032] Figure 1 This is a flowchart illustrating a distributed gridded data processing method provided in Embodiment 1 of the present invention. This embodiment is applicable to situations requiring accelerated computation of complex business processes. The method can be executed by a distributed gridded data processing device, which can be implemented in hardware and / or software and configured in an electronic device with data processing capabilities. Figure 1 As shown, the method includes:

[0033] S110, the processing logic of the data to be processed is obtained through a preset rule engine, and the data to be processed is split into an atomic operation queue and a parameter queue; wherein, the atomic operation queue contains the dependency relationship between each atomic operation.

[0034] The rule engine can be a component embedded in an application. It can separate business decisions from application code and use predefined semantic modules to write those decisions. For example, in this embodiment, the default rule engine can be a component that reads the complete processing logic of the data to be processed, breaks it down into atomic operations, and initializes the dependencies between these atomic operations.

[0035] The data to be processed can be a single complex computation request waiting to be processed. In this embodiment, it can be broken down into atomic operations using a preset rule engine. An atomic operation can be the smallest single instruction or operation that cannot be further broken down by the processor.

[0036] The processing logic can be the dependencies between atomic operations in the data to be processed. For example, there is a dependency between atomic operation A and the computation result of atomic operation B.

[0037] In this embodiment, optionally, the process of obtaining the processing logic of the data to be processed through a preset rule engine can be as follows: inputting each operation logic to be processed into a unified atomic operation entry point, establishing a structured log collection device, embedding each atomic operation, and collecting the input parameters, output parameters, and execution order information of each operation. The structured log can be a log that follows a certain structure, where each line of log represents a structure.

[0038] A queue is a special type of linear list that only allows deletion operations at the front of the list and insertion operations at the back. An atomic operation queue can be a linear list composed of a series of atomic operations. A parameter queue can be a linear list composed of the input parameters, output parameters, and execution order information of each atomic operation.

[0039] In this embodiment, optionally, the process of splitting the data to be processed into an atomic operation queue and a parameter queue can be: performing pattern matching and splitting of complex logic processing by loading configuration items, reorganizing fine-grained atomic operations, etc. For example, a segmentation device based on semantic analysis further decomposes the original atomic operations into four types of fine-grained atomic operations: CRUD operations, basic operations, scientific computing, and set operations; it also removes independent fine-grained steps in the original atomic operations that do not have complex dependencies, parallelizes the extracted computational operations, simplifies serial logic, and prioritizes the execution order of dependent atomic operations to prevent task blocking.

[0040] The dependencies between atomic operations can be parameter dependencies between atomic operations. For example, if atomic operation B must be performed after atomic operation A is completed, we define this relationship as a "dependency" and denoted by the symbol "A–>D", indicating that operation D depends on operation A.

[0041] In this embodiment, by subdividing complex logic processing operations, the improved atomic operations are well adapted to acceleration solutions including grid technology, cloud computing, and multi-threaded computing.

[0042] S120, calculate the time cost of each atomic operation in the atomic operation queue using a time calculator.

[0043] The time trial calculator can be a device that calculates the computational complexity and time cost of atomic operations to determine the computation time of atomic operations. If the time trial calculator attempts atomic operation A and finds it to be very time-consuming, then a high-performance computing node is allocated. A high-performance computing node can be a computing node in a server with hardware configurations and peak computing speeds (PSS) significantly superior to ordinary equipment. Peak computing speed (PSS) can be the number of floating-point operations performed per second, used to estimate computer processing speed or performance. The time cost of an atomic operation can be the time taken by the computing node to execute the atomic operation.

[0044] In this embodiment, optionally, the process of calculating the time cost of each atomic operation in the atomic operation queue using a time calculator can be: directly calling the time calculation result of the atomic operation matching the pre-calculation operation as the time cost, or performing a calculation through a calculation node and using the calculated time as the time cost. The pre-calculation operation can be a pre-stored calculation operation of the atomic operation. The calculation node can be a node that calculates the calculation time of the atomic operation.

[0045] In this embodiment, time cost calculations are performed on each atomic operation to obtain time cost data. Then, by utilizing the characteristics of grid technology, subtasks with higher complexity and larger time cost calculation results are allocated to high-performance device nodes in the grid system. This achieves the goal of minimizing the processing time difference between nodes after grid computing, avoiding the increase in total time due to uneven task allocation.

[0046] S130, based on the time cost of each atomic operation and the processing capacity of the distributed grid computing nodes, the atomic operations in the atomic operation queue are distributed according to the dependency relationship.

[0047] Grid technology, a type of distributed technology, is a new technology primarily designed for processing complex scientific tasks. It utilizes the internet to connect computers geographically dispersed across different locations into a virtual supercomputer. Each computer participating in processing large-scale tasks is a node in the grid, and the entire processing flow consists of a grid composed of numerous nodes.

[0048] The processing power of distributed grid computing nodes can be categorized by their ability to handle atomic operations and the time complexity required for complex logical processing. Computing nodes can be classified into high-performance dedicated device nodes and general-purpose device nodes based on their processing power. Dedicated device nodes are computing nodes located in servers with significantly superior hardware configurations and peak computing speeds per second compared to ordinary devices. Examples include high-performance computers with 32-core high-frequency processors and 128GB of RAM, or graphics cards used for computer vision or deep learning tasks. General-purpose device nodes are computing nodes widely used in production environments and shared by most systems within cloud computing resources. The mainstream configuration for general-purpose nodes is 4 cores and 8GB of RAM, suitable for general transactions and with low maintenance costs. However, due to limitations in memory and computing power, their performance may not meet real-time requirements when performing complex logical processing.

[0049] In this embodiment, optionally, the process of distributing atomic operations in the atomic operation queue based on the dependencies, according to the time cost of each atomic operation and the processing capacity of the distributed grid computing nodes, can be as follows: Based on a greedy algorithm and time trial calculations, processing is performed according to the principle of minimizing total time cost. Each subtask is allocated to each task node according to the principle of balancing the execution time of each node. A greedy algorithm always makes the best choice at the moment when solving a problem. That is, it does not consider the overall optimal solution; it only makes a locally optimal solution in some sense. Greedy algorithms do not obtain globally optimal solutions for all problems, but they can produce globally optimal solutions or approximate solutions for many problems across a wide range.

[0050] In this embodiment, dedicated equipment and standard equipment are incorporated into the gridded technology system. Combined with the equipment load monitoring device and the task distribution device, the equipment load is monitored in real time. When the load level of the dedicated equipment is low, the most time-consuming sub-task ("bottleneck" task) is preferentially allocated to the dedicated equipment incorporated into the gridded technology system. This achieves the effects of reducing the idle rate of dedicated equipment resources, accurately controlling the usage intensity of dedicated equipment, saving dedicated equipment resources, reducing the time consumption of complex sub-tasks, and improving the real-time performance of the system.

[0051] S140, Obtain the processing results of the distributed grid computing nodes, and merge them based on the dependency relationship to obtain the distributed grid processing results.

[0052] The processing result of the distributed grid computing node can be the result of the distributed grid computing node performing the assigned atomic operations.

[0053] In this embodiment, the processing results are obtained by sending task instructions and parameters to each computing node in parallel.

[0054] The result of distributed grid processing can be calculated by each node in the grid system returning intermediate results to the memory result set and then calculating the final result.

[0055] In this embodiment, after obtaining the processing results from the gridded computing nodes, they are aggregated through a predefined intermediate variable queue. The final result is calculated after each node in the grid system returns intermediate results to the memory result set. The intermediate variable queue can be a transitional queue. For example, in this embodiment, the intermediate variable queue can be a queue for aggregating the memory results processed by the gridded computing nodes.

[0056] In this embodiment, by breaking down the computational tasks of each scenario into their own computational logic in a fine-grained manner, and adding their sub-logic, elements, and computation results to reusable information storage, it is possible to reuse the same processing device for each computational task by adopting a common computational method.

[0057] The technical solution provided in this application, by splitting the data to be processed and adapting to the grid technology acceleration scheme, and by calculating the time overhead of atomic operations and the processing capabilities of distributed grid computing nodes, can reasonably allocate atomic operations and achieve the effect of significantly improving the real-time performance of the system.

[0058] Example 2

[0059] Figure 2 This is a flowchart of a distributed gridded data processing method provided in Embodiment 2 of the present invention. This embodiment is based on the above embodiment and optimized. Specifically, the optimization is as follows: the time cost of each atomic operation in the atomic operation queue is calculated by a time calculator, including: identifying the matching result between each atomic operation and a pre-cached pre-calculation operation; for the time calculation result of the atomic operation that matches the pre-calculation operation, the cached calculation result is used as the time cost; for the unmatched atomic operation, a calculation is performed through a calculation node, and the calculation time is used as the time cost. Figure 2 As shown, the method in this embodiment specifically includes the following steps:

[0060] S210, the processing logic of the data to be processed is obtained through a preset rule engine, and the data to be processed is split into an atomic operation queue and a parameter queue; wherein, the atomic operation queue contains the dependencies between atomic operations.

[0061] In this embodiment, optionally, the data to be processed is split into an atomic operation queue and a parameter queue, including: parsing the task instructions and parameters of the data to be processed into an abstract syntax tree; and rewriting the task instructions and parameters into a shardable atomic operation queue and parameter queue based on the abstract syntax tree. The task instructions can be commands that instruct the computer to perform a certain operation, such as calculating the valuation of this batch of bonds. The abstract syntax tree can be an abstract representation of the source code syntax structure. It represents the syntax structure of a programming language in a tree-like form, where each node in the tree represents a structure in the source code.

[0062] In this implementation, the task instructions and parameters of the data to be processed are rewritten into a shardable atomic operation queue and parameter queue through an abstract syntax tree. This allows the split atomic operation queue and parameter queue to be well adapted to acceleration solutions including grid technology, cloud computing, and multi-threaded computing.

[0063] In this embodiment, optionally, the data to be processed includes basic CRUD operations, computationally intensive operations, and basic mathematical and aggregation operations of the database. Correspondingly, based on the abstract syntax tree, the task instructions and parameters are rewritten into a shardable atomic operation queue and parameter queue, including: using a first identifier for basic CRUD operations, a second identifier for computationally intensive operations, and a third identifier for basic mathematical and aggregation operations; using the first, second, and third identifiers, nested logic is represented as an atomic operation queue; or, the judgment logic for nested computationally intensive operations is represented as an atomic operation queue selection; the atomic operation queue and related variable queue are stored to obtain the splitting results of the atomic operation queue and parameter queue. Here, computationally intensive operations can be operations with high computational load and where computational speed is the primary metric. Aggregation operations can be operations that calculate a value from a set of values. Sharding can divide a large database into smaller, faster, and easier-to-manage parts; for example, in this embodiment, sharding can divide the data to be processed into atomic operation queues and parameter queues using a processor.

[0064] In this embodiment, by subdividing complex logic processing operations and rewriting task instructions and parameters into distributed executable task instructions and parameters, the goal of adapting a single complex logic processing task to parallelization acceleration solutions such as grid technology is achieved.

[0065] S220 identifies the matching results between each atomic operation and the pre-cached pre-computed operation.

[0066] In this embodiment, the process of identifying the matching results between each atomic operation and the pre-cached pre-calculation operations can be as follows: A relational storage system is established in the cache server to record computational element information, including logic, parameters, and underlying methods. Matching is prioritized during computation task execution. The cache server can be a server that stores information about accessed objects. It stores frequently accessed objects in a system closer to the user, thus speeding up access to these objects again.

[0067] S230, for the time calculation result of the atomic operation that matches the pre-calculation operation, the time cost is the cached calculation result.

[0068] The pre-calculation operation can be a pre-stored trial operation of the atomic operation. The time overhead can be the time taken by the computing node to execute the atomic operation.

[0069] S240: For unmatched atomic operations, trial calculations are performed through trial nodes, and the trial calculation time is used as the time cost.

[0070] Among them, the trial node can be a node that calculates the trial time of atomic operations.

[0071] In this example, by calculating the time cost of each atomic operation, the time cost data is used for precise task allocation, thus avoiding the increase in total time due to uneven task allocation.

[0072] In this embodiment, optionally, after performing trial calculations on unmatched atomic operations through trial calculation nodes and using the calculation time as the time cost, the method further includes: storing the time cost of the unmatched atomic operations in a cache as the calculation result of the pre-trial calculation operation; correspondingly, the method further includes: if an update cycle is detected, destroying the cached calculation result of the pre-trial calculation operation; or, if an update cycle is detected, recalculating the cached calculation result of the pre-trial calculation operation to update the cached calculation result of the pre-trial calculation operation.

[0073] In this example, by performing calculations and updating the time taken for each atomic operation, and updating the cached values ​​in the cache server, the real-time performance and accuracy of the calculation results can be achieved.

[0074] S250, based on the time cost of each atomic operation and the processing capacity of the distributed grid computing nodes, the atomic operations in the atomic operation queue are distributed according to the dependency relationship.

[0075] In this embodiment, optionally, the construction process of the distributed grid computing node includes: acquiring general-purpose resources and dedicated resources to be accessed; constructing the distributed grid computing node based on the general-purpose resources and the dedicated resources; wherein the general-purpose resources are constructed in the form of cloud resources, and the dedicated resources are constructed in the form of independent clusters. The general-purpose resources can be cloud computing equipment resources that are widely used in production environments and shared by the vast majority of systems. The dedicated resources can be servers with hardware configurations and peak computing speeds per second that are significantly superior to ordinary equipment. Peak computing speed per second can be the number of floating-point operations performed per second, commonly used to estimate computer computing speed or computer performance. A cluster can be a group of computers that, as a whole, provide a set of network resources to the user.

[0076] In this embodiment, by incorporating cloud computing resources of dedicated equipment and standard equipment into the grid technology system, and leveraging the characteristics of cloud computing to achieve reasonable task allocation, the system aims to reduce the idle rate of dedicated equipment resources, precisely control the intensity of dedicated equipment usage, and save dedicated equipment resources.

[0077] In this embodiment, optionally, before distributing the atomic operations in the atomic operation queue based on the dependency relationship according to the time cost of each atomic operation and the processing capacity of the distributed grid computing nodes, the method further includes: determining the load label of each distributed grid computing node through a preset load reporting rule and caching the load label; obtaining the hardware allocation result of the distributed grid computing node and calling the load label; determining the processing capacity of the distributed grid computing node according to the hardware allocation result and the load label. The load reporting rule can be the computation time of the atomic operations processed by the load, and the load label can include dedicated device load or general load.

[0078] In this embodiment, by using a real-time monitoring device load mechanism, when the load level of the dedicated device is low, the allocation ratio of each sub-task is further tilted towards the dedicated device, thereby achieving the effect of further compressing the task execution time and reducing the idle rate of the dedicated device.

[0079] S260, Obtain the processing results of the distributed grid computing nodes, and merge them based on the dependency relationship to obtain the distributed grid processing results.

[0080] In this embodiment, by decomposing the computational tasks of each scenario into their own computational logic in a fine-grained manner, and by adding their sub-logic, elements, and computation results to reusable information storage, it is possible to reuse the same processing device for each computational task by adopting a common computational method.

[0081] The technical solution provided in this application identifies the matching results between each atomic operation and the pre-cached pre-calculation operation. It determines the time cost by directly calling the time calculation result of the atomic operation that matches the pre-calculation operation or by performing calculation through the calculation node. Combined with the equipment load monitoring device and the task distribution device, it achieves the effect of matching high and low equipment configurations, controlling costs, reducing the time consumption of complex sub-tasks, and greatly improving the real-time performance of the system through reasonable task allocation.

[0082] Example 3

[0083] This embodiment is a preferred embodiment provided based on the two embodiments described above. Figure 3 This is a technical solution system diagram provided according to Embodiment 3 of the present invention, such as... Figure 3 As shown, the technical solution of this embodiment is divided into the following 6 parts;

[0084] 1) Establish a grid system

[0085] Both general-purpose and specialized resources are incorporated into the grid-based technology system. The grid system includes: general-purpose resources provided in the form of cloud resources, specialized resources provided in the form of independent on-premises clusters, application servers, and shared cache servers.

[0086] In this embodiment, by introducing a grid-based technology system and incorporating both standard and specialized equipment, the system achieves the effects of combining high and low-end equipment configurations, controlling costs, and reducing the time consumption of complex sub-tasks.

[0087] 2) Computational logic processing

[0088] The complete processing logic is read by the rules engine, broken down into atomic operations, and the dependencies between each atomic operation are initialized. After optimization, the results are used by the subsequent time calculator and task dispatcher.

[0089] The process of initializing the dependencies between atomic operations can be as follows: input the logic of each operation to be processed into a unified atomic operation entry point, establish a structured log collection device, embed it into each atomic operation, and collect the input parameters, output parameters, and execution order information of each operation. The syntax optimizer obtains the parameter dependencies of each operation in the form of X->Y through semantic analysis, removes operations O that do not have dependencies, and prioritizes the execution order of the atomic operations X that are depended upon in each dependency relationship.

[0090] In this embodiment, the dependencies between atomic operations are initialized by constructing a network diagram of the correspondence between processing logics. At the same time, a fine-grained atomic operation execution queue is used to realize a pluggable computing entry point that is universal for various business scenarios, so that the original serial computing tasks are transformed into parallel subtasks and adapted to the grid technology system.

[0091] 3) Equipment load monitoring

[0092] Configure a dedicated device low-load flag on the cache server, periodically retrieve monitoring data, and update the flag. In subsequent steps, the device low-load flag will be used as the basis for task routing.

[0093] In this embodiment, by introducing a real-time monitoring mechanism for equipment load, when the load level of the dedicated equipment is low, the allocation ratio of each sub-task is further tilted towards the dedicated equipment, thereby achieving the effect of further compressing the task execution time and reducing the idle rate of the dedicated equipment.

[0094] 4) Time Trial Batch Processing

[0095] The time calculator is used to calculate the computational complexity and time cost of each subtask in the network diagram. In addition to being called directly, the time calculator also performs periodic batch processing to calculate and update the time consumption of each atomic operation and update the cache values ​​in the cache server.

[0096] One approach to identifying the same computational task that can be directly invoked is to establish a relational storage in the cache server to record computational element information, including logic, parameters, and underlying methods, and to prioritize matching when the computational task is executed.

[0097] Whether to assign a high-performance computing node or a general-purpose computing node to an atomic operation depends primarily on the time taken by the timer. If the timer attempts atomic operation A and finds it to be very time-consuming, then a high-performance computing node is assigned. This result is stored in the timer cache server to record the time taken for each atomic operation. However, this time is not static; updates may affect it. Therefore, it needs to be periodically repeated and updated.

[0098] In this embodiment, time cost calculations are performed on the actual content of each sub-step to obtain time cost data. Then, by utilizing the characteristics of grid technology, sub-tasks with high complexity and large time cost calculation results are allocated to high-performance device nodes in the grid system. This achieves the goal of minimizing the processing time difference between each node after grid computing, and avoids the phenomenon of increased total time due to uneven task allocation.

[0099] 5) Task Distribution

[0100] The computing interfaces deployed on dedicated and general-purpose devices are used to call these devices within the grid system to perform atomic operations and obtain results. The distribution mechanism is based on a greedy algorithm, using time estimation results as a basis and processing tasks according to the principle of minimizing total time overhead. Subtasks are precisely allocated to each task node based on the principle of balancing execution time across nodes. Considering that dedicated devices have significantly higher processing speeds than standard devices, the task distribution device allocates subtasks to dedicated devices in descending order of estimated time, prioritizing "bottleneck" tasks whose time estimation results or time percentages exceed a threshold for allocation to dedicated devices.

[0101] Among them, the greedy algorithm can be the global optimal solution to the problem that can be obtained through a series of locally optimal choices. In other words, when considering what choice to make, we only consider the best choice for the current problem and do not consider the results of subproblems.

[0102] In this embodiment, the time calculation results of atomic operations are sorted from largest to smallest according to time. High-performance computing nodes are allocated first for those with larger calculation times, while general computing nodes are allocated for those with smaller calculation times. This allocation ensures that the overall time of the task is minimized.

[0103] 6) Summary of Results

[0104] After obtaining the in-memory result set returned by the gridding process, it is aggregated through a predefined intermediate variable queue. The queue, split by the computational logic splitting device, can be directly executed by the rule engine to obtain the results. After waiting for each node in the grid system to return intermediate results to the in-memory result set, the final result is calculated and output.

[0105] The calculation results of the specialized equipment can be stored in a cache server as needed, which solves the problem that in specific scenarios, the calculation rules are fixed but the required formulas are numerous and the amount of calculation is large.

[0106] In this embodiment, by opening up public atomic operation entry points and a unified logical information and element information storage system, an acceleration device for complex data processing tasks with strong reusability is provided to the outside world.

[0107] Specifically, the computational logic processing includes a sub-process, which uses the aforementioned modules in collaboration to complete the proxy function of this solution:

[0108] I. Task Breakdown Process

[0109] (2.1) Obtain the user's input task instructions and parameters by parsing HTTP request packets and structured logs;

[0110] (2.2) The task instructions and parameters are parsed into an abstract syntax tree by the lexical analyzer and the syntax analyzer, and the information required for resource fragmentation is extracted;

[0111] (2.3) Rewrite the task instructions and parameters into distributed executable task instructions and parameters;

[0112] The atomic operations contained in the rewrite results are mainly divided into (a) basic database operations, labeled CRUD; (b) computationally intensive operations, labeled CAL; and (c) basic mathematical and aggregate operations, labeled MATH.

[0113] The rewriting logic includes:

[0114] (i) Nested logic of atomic operations in the form of "MATH(CAL(param_x),CAL(param_y),CAL(param_z))" is processed into a queue data structure of [var1=CAL(param_x); var2=CAL(param_y); var3=CAL(param_z); MATH(var1,var2,var3)]. In this scenario, the specific operations involved in CAL include, but are not limited to: linear regression calculation (LogisticRegression), LM nonlinear regression calculation (LMRegression), machine learning backpropagation calculation (backPropagation), matrix multiplication calculation (matMul); the specific operations involved in MATH include, but are not limited to: summation calculation (sum), date difference calculation (dateDiff);

[0115] (ii) Process the nested computationally intensive conditional logic, such as “if(CAL(param_x)){...}else if(CAL(param_y){})”, into [flag1 = CAL(param_x); flag2 = CAL(param_y); if(flag1){...}else if(flag2){}];

[0116] (iii) Store the processed variable names in the cache server by generating random strings. During the processing of the operation queue, store and read the calculated variable values ​​in the cache server. When the processing queue is empty, clear the cache of the relevant variables.

[0117] After splitting the operations into three categories according to (i), (ii), and (iii) above, this invention splits the original operations into a finer-grained atomic operation queue, OperationQueue, and variable storage queue, and can support task distribution and parallel processing in the form of grid technology.

[0118] The resource sharding key can be the address of the resource processor to be allocated when allocating resources for an atomic operation. However, directly allocating addresses is prone to errors, so it is stored in a format similar to a Map<'key', memory address>.

[0119] For example: Method 1: Directly allocate memory resources in the range of 0000000001-0000FFFFF for the atomic operation A.

[0120] Method 2: Create a resource sharding map, assigning a 'resource sharding key' of 'a' to the region of resource 0000000001-0000FFFFF.<a,0000000001-0000FFFFF>

[0121] Allocate a 'resource shard key' of region 0000FFFFF-0001FFFFF with Map.<b,0000FFFFF-0001FFFFF>

[0122] When allocating resources to A, it is not necessary to tell it the specific memory address range of 0000000001-0000FFFFF. It is only necessary to tell the system that the resource sharding key is a, and the system will search in the resource sharding map to find the corresponding resource address range of 0000000001-0000FFFFF.

[0123] (2.4) Match processor resource sharding keys for each operation in the atomic operation queue according to the user-preset algorithm, and calculate the routing path according to the matched processor;

[0124] (2.5) Send task instructions and parameters to each data node in parallel, and the execution engine is responsible for balancing the connection pool and memory resources;

[0125] (2.6) Perform streaming or full memory result set merging calculations based on the AST;

[0126] (2.7) Encapsulate the memory result set and return it to the client.

[0127] This solution, through its configuration, constructs a network diagram of the correspondence between processing elements and processing logic, and a fine-grained atomic operation execution queue. This enables a universal, pluggable computing entry point across various business scenarios, transforming traditional serial computing tasks into parallel subtasks, thus adapting to gridded technology systems. Simultaneously, by introducing gridded technology and incorporating both standard and dedicated equipment, combined with equipment load monitoring and task distribution devices, it reduces dedicated equipment resource idle rates through reasonable task allocation, precisely controls dedicated equipment usage intensity, conserves dedicated equipment resources, and achieves the effects of high-low equipment configuration, cost control, and reduced time consumption for complex subtasks. Furthermore, by opening a public atomic operation entry point and a unified logical and element information storage system, this solution provides a highly reusable acceleration device for complex data processing tasks.

[0128] This embodiment is applicable to low-latency scenarios requiring accelerated processing of complex scientific calculations, including bond price and yield problems related to time-series information, deep learning inference tasks such as text classification, regression problems such as logistic regression, and acceleration of Levenid-Marquardt nonlinear regression. This embodiment provides a specific scenario for calculating the full price of a bond. Figure 4 This is a network diagram of bond price yield in the bond full price calculation scenario provided by Embodiment 3 of the present invention.

[0129] The first step is to construct the underlying assets for which the full price of bonds needs to be calculated, forming a computational element part of reusable information;

[0130] The second step is to construct the known full-price formulas. For example, different bond types such as discount bonds, lump-sum principal and interest repayment bonds, ordinary fixed-rate bonds, floating-rate bonds, short-term U.S. Treasury bonds, and long-term U.S. Treasury bonds use different formulas. These formulas are then processed by the computational logic processing device to form the processing logic part of reusable information.

[0131] The third step involves constructing fine-grained atomic operation information based on the correspondence between different bonds and yield formulas, forming an operation queue that can adapt to grid technology, and then performing calculations and outputs.

[0132] The fourth step involves adding new bond types and valuation formulas to the processing logic. This only requires adding some bond type information and valuation formula logic to the existing information. Based on different conditions, corresponding logic and atomic operation execution queues are constructed, and the rest can be reused by reading cached reusable information.

[0133] After the process is completed, reusable information related to the full price of bonds, consisting of pluggable sub-logic and elements, and calculation results, will be generated and stored on demand in the cache server. Each computational task in each scenario can decompose its own computational logic in a fine-grained manner through the computational logic processing device, adding its own sub-logic and elements to the reusable information cache. Therefore, in this embodiment, each computational task can reuse the same set of reusable information generation and acceleration functions.

[0134] Example 4

[0135] Figure 5 This is a schematic diagram of a distributed gridded data processing device according to Embodiment 4 of the present invention. This device can execute the distributed gridded data processing method provided in any embodiment of the present invention, and possesses the corresponding functional modules and beneficial effects for executing the method. Figure 5 As shown, the device includes:

[0136] The data splitting module 510 is used to obtain the processing logic of the data to be processed through a preset rule engine, and split the data to be processed into an atomic operation queue and a parameter queue; wherein, the atomic operation queue contains the dependencies between each atomic operation;

[0137] The time cost calculation module 520 is used to calculate the time cost of each atomic operation in the atomic operation queue through a time calculator.

[0138] The atomic operation distribution module 530 is used to distribute each atomic operation in the atomic operation queue based on the dependency relationship, according to the time cost of each atomic operation and the processing capability of the distributed grid computing nodes.

[0139] The processing result acquisition module 540 is used to acquire the processing results of the distributed grid computing nodes and merge them based on the dependency relationship to obtain the distributed grid processing results.

[0140] Based on the above technical solutions, optionally, the device further includes: a computing node construction module, comprising:

[0141] The resource acquisition unit is used to acquire general and special resources to be accessed.

[0142] A computing node construction unit is used to construct distributed gridded computing nodes based on the general resources and the special resources; wherein the general resources are constructed in the form of cloud resources, and the special resources are constructed in the form of independent clusters.

[0143] Based on the above technical solutions, optionally, the time cost calculation module includes:

[0144] The atomic operation identification unit is used to identify the matching results between each atomic operation and the pre-cached pre-calculated operations.

[0145] The trial calculation result retrieval unit is used to call the time trial calculation result of the atomic operation that matches the pre-cached pre-trial calculation operation if the atomic operation matches the pre-cached pre-trial calculation operation, with the cached trial calculation result as the time cost.

[0146] The trial calculation result calculation unit is used to perform trial calculations on the unmatched atomic operations through the trial calculation node if the atomic operation does not match the pre-cached pre-trial calculation operation, and the calculation time is used as the time cost.

[0147] Based on the above technical solutions, optionally, the device further includes:

[0148] The trial calculation result storage module is used to store the time overhead of the unmatched atomic operations in a cache as the trial calculation result of the pre-trial calculation operation.

[0149] The trial calculation result destruction module will destroy the cached trial calculation results if it detects that the update cycle has been reached.

[0150] If the calculation result update module detects that the update cycle has been reached, it will recalculate the calculation results of the cached pre-calculation operations to update the cached pre-calculation operation results.

[0151] Based on the above technical solutions, optionally, the device further includes: a computing node processing capability determination module, specifically including:

[0152] The load label determination unit is used to determine the load label of each distributed grid computing node according to a preset load reporting rule, and cache the load label;

[0153] The load tag allocation unit is used to obtain the hardware allocation results of distributed grid computing nodes and to invoke load tags;

[0154] The computing node processing capacity determination unit is used to determine the processing capacity of the distributed grid computing nodes based on the hardware allocation results and the load labels.

[0155] Based on the above technical solutions, optionally, the data splitting module 510 includes:

[0156] The data parsing unit is used to parse the task instructions and parameters of the data to be processed into an abstract syntax tree;

[0157] The task instruction and parameter rewriting unit is used to rewrite the task instructions and parameters into a shardable atomic operation queue and parameter queue based on the abstract syntax tree.

[0158] Based on the above technical solutions, optionally, the data to be processed includes basic CRUD operations, computationally intensive operations, and basic mathematical and aggregation operations in the database.

[0159] Accordingly, the task instruction and parameter rewriting unit includes:

[0160] The data to be processed identification subunit uses the first identifier for basic CRUD operations in the database, the second identifier for computationally intensive operations, and the third identifier for basic mathematical operations and aggregation operations.

[0161] A sub-unit is generated in the form of an atomic queue, which is used to represent nested logic as an atomic operation queue using the first identifier, the second identifier, and the third identifier; or, to represent the judgment logic of nested computationally intensive operations as an atomic operation queue selection.

[0162] The splitting results generate sub-units for storing the atomic operation queue and related variable queue, thus obtaining the splitting results of the atomic operation queue and parameter queue.

[0163] The distributed gridded data processing device provided in this embodiment of the invention can execute the distributed gridded data processing method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0164] Example 5

[0165] Embodiment 5 of the present invention provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a distributed grid-based data processing method as provided in all embodiments of the present application:

[0166] The processing logic of the data to be processed is obtained through a preset rule engine, and the data to be processed is split into an atomic operation queue and a parameter queue; wherein, the atomic operation queue contains the dependencies between the atomic operations;

[0167] The time cost of each atomic operation in the atomic operation queue is calculated using a time calculator.

[0168] Based on the time cost of each atomic operation and the processing capacity of the distributed grid computing nodes, each atomic operation in the atomic operation queue is distributed according to the dependency relationship.

[0169] Obtain the processing results of the distributed grid computing nodes, and merge them based on the dependencies to obtain the distributed grid processing results.

[0170] Any combination of one or more computer-readable media may be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0171] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including—but not limited to—electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0172] The program code contained on a computer-readable medium may be transmitted using any suitable medium, including—but not limited to—wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0173] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0174] Example 6

[0175] Embodiment 6 of this application provides an electronic device. Figure 6 This is a schematic diagram of the structure of an electronic device according to Embodiment Six of the present invention. Figure 6 As shown, this embodiment provides an electronic device 600, which includes: one or more processors 620; and a storage device 610 for storing one or more programs. When the one or more programs are executed by the one or more processors 620, the one or more processors 620 implement the distributed grid processing method for data provided in this application embodiment. The method includes:

[0176] The processing logic of the data to be processed is obtained through a preset rule engine, and the data to be processed is split into an atomic operation queue and a parameter queue; wherein, the atomic operation queue contains the dependencies between the atomic operations;

[0177] The time cost of each atomic operation in the atomic operation queue is calculated using a time calculator.

[0178] Based on the time cost of each atomic operation and the processing capacity of the distributed grid computing nodes, each atomic operation in the atomic operation queue is distributed according to the dependency relationship.

[0179] Obtain the processing results of the distributed grid computing nodes, and merge them based on the dependencies to obtain the distributed grid processing results.

[0180] Of course, those skilled in the art will understand that the processor 620 also implements the technical solution of the distributed grid processing method for data provided in any embodiment of this application.

[0181] Figure 6 The electronic device 600 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0182] like Figure 6 As shown, the electronic device 600 includes a processor 620, a storage device 610, an input device 630, and an output device 640; the number of processors 620 in the electronic device can be one or more. Figure 6 Taking a processor 620 as an example; the processor 620, storage device 610, input device 630, and output device 640 in the electronic device can be connected via a bus or other means. Figure 6 Taking the connection between China and Israel via bus 650 as an example.

[0183] Storage device 610, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and module units, such as the program instructions corresponding to the flow control method of the distributed system in the embodiments of this application.

[0184] Storage device 610 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created based on terminal usage. Furthermore, storage device 610 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, storage device 610 may further include memory remotely located relative to processor 620, and these remote memories can be connected via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0185] Input device 630 can be used to receive input digital, character, or voice information, and to generate key signal inputs related to user settings and function control of the electronic device. Output device 640 may include electronic devices such as a display screen and a speaker.

[0186] The electronic device provided in this application embodiment can adapt to the grid technology system by constructing a network diagram of the correspondence between processing elements and processing logic and a fine-grained atomic operation execution queue. It distributes each atomic operation through a time calculation device and a task distribution device, thereby achieving the effects of high and low equipment configuration, cost control, and reduction of complex sub-task time consumption, which greatly improves the real-time performance of the system.

[0187] The flow control device, medium, and electronic equipment of the distributed system provided in the above embodiments can execute the distributed gridding data processing method provided in any embodiment of this application, and have the corresponding functional modules and beneficial effects of executing the method. Technical details not described in detail in the above embodiments can be found in the distributed gridding data processing method provided in any embodiment of this application.

[0188] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0189] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for distributed grid processing of data, characterized by, The method comprises: obtaining processing logic of to-be-processed data through a preset rule engine, and splitting the to-be-processed data into an atomic operation queue and a parameter queue; wherein the atomic operation queue comprises a dependency relationship between each atomic operation, and the dependency relationship is a parameter dependency relationship between each atomic operation; calculating time consumption of each atomic operation in the atomic operation queue through a time trial calculator; distributing each atomic operation in the atomic operation queue based on the dependency relationship according to the time consumption of each atomic operation and processing capability of a distributed grid computing node; obtaining a processing result of the distributed grid computing node, and merging to obtain a distributed grid processing result based on the dependency relationship; wherein the splitting of the to-be-processed data into the atomic operation queue and the parameter queue comprises: parsing task instructions and parameters of the to-be-processed data into an abstract syntax tree; rewriting the task instructions and parameters into an atomic operation queue and a parameter queue that can be processed in slices based on the abstract syntax tree; wherein the to-be-processed data comprises database basic insert, delete, update and query operations, computation-intensive operations, and basic mathematical operations and aggregation operations; correspondingly, the rewriting of the task instructions and parameters into the atomic operation queue and the parameter queue that can be processed in slices based on the abstract syntax tree comprises: the database basic insert, delete, update and query operations adopt a first identifier, the computation-intensive operations adopt a second identifier, and the basic mathematical operations and aggregation operations adopt a third identifier; using the first identifier, the second identifier and the third identifier, nested logic is expressed in the form of the atomic operation queue, or the judgment logic of nested computation-intensive operations is expressed in the form of atomic operation queue selection; storing the atomic operation queue and a related variable queue to obtain a splitting result of the atomic operation queue and the parameter queue.

2. The method of claim 1, wherein, The construction process of the distributed grid computing node comprises: obtaining general resources and special resources to be accessed; constructing a distributed grid computing node based on the general resources and the special resources; wherein the general resources are constructed in the form of cloud resources, and the special resources are constructed in the form of independent clusters.

3. The method of claim 1, wherein, calculating the time consumption of each atomic operation in the atomic operation queue through the time trial calculator comprises: identifying a matching result between each atomic operation and a pre-cached pre-trial operation; for the time trial result of the atomic operation matched with the pre-trial operation, using the cached trial result as the time consumption; for the unmatched atomic operation, performing a trial through a trial node, and taking the trial time consumption as the time consumption.

4. The method of claim 3, wherein, after performing the trial through the trial node for the unmatched atomic operation and taking the trial time consumption as the time consumption, the method further comprises: storing the time consumption of the unmatched atomic operation into the cache as a trial result of a pre-trial operation; correspondingly, the method further comprises: if it is detected that an update period is reached, destroying the trial result of the cached pre-trial operation; or, if it is detected that an update period is reached, re-trialing the trial result of the cached pre-trial operation to update the trial result of the cached pre-trial operation.

5. The method of claim 1, wherein, Before distributing each atomic operation in the atomic operation queue based on the dependency relationship according to the time overhead of each atomic operation and the processing capacity of the distributed grid computing node, the method further comprises: determining the load label of each distributed grid computing node through a preset load reporting rule, and caching the load label; obtaining the hardware allocation result of the distributed grid computing node, and calling the load label; determining the processing capacity of the distributed grid computing node according to the hardware allocation result and the load label.

6. A distributed grid processing apparatus of data, characterized by, Comprise: a to-be-processed data splitting module configured to obtain the processing logic of to-be-processed data through a preset rule engine, and split the to-be-processed data into an atomic operation queue and a parameter queue; wherein the atomic operation queue comprises the dependency relationship between each atomic operation, and the dependency relationship is the parameter dependency relationship between each atomic operation; a time overhead calculation module configured to calculate the time overhead of each atomic operation in the atomic operation queue through a time trial calculator; an atomic operation distribution module configured to distribute each atomic operation in the atomic operation queue based on the dependency relationship according to the time overhead of each atomic operation and the processing capacity of the distributed grid computing node; a processing result obtaining module configured to obtain the processing result of the distributed grid computing node, and merge to obtain a distributed grid processing result based on the dependency relationship; wherein the to-be-processed data splitting module comprises: a to-be-processed data parsing unit configured to parse the task instruction and parameters of the to-be-processed data into an abstract syntax tree; a task instruction and parameter rewriting unit configured to rewrite the task instruction and parameters into an atomic operation queue and a parameter queue that can be processed in slices based on the abstract syntax tree; wherein the to-be-processed data comprises database basic insert, delete, update and query operations, computation-intensive operations, and basic mathematical operations and aggregation operations; Correspondingly, the task instruction and parameter rewriting unit comprises: a to-be-processed data identification subunit configured to use a first identification for database basic insert, delete, update and query operations, a second identification for computation-intensive operations, and a third identification for basic mathematical operations and aggregation operations; an atomic queue form generation subunit configured to use the first identification, the second identification and the third identification to represent nested logic in the form of an atomic operation queue, or to represent the judgment logic of nested computation-intensive operations in the form of atomic operation queue selection; a splitting result generation subunit configured to store the atomic operation queue and the related variable queue to obtain the splitting result of the atomic operation queue and the parameter queue.

7. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the distributed grid processing method of data according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing the processor to implement the distributed meshing processing method of the data according to any one of claims 1-5 when executed.

Images