A distributed computing load balancing method and system

Abstract

The embodiments of this specification disclose a distributed computing load balancing method and system. The method includes receiving multiple computing tasks that can be executed in parallel divided from a target task, reading the code of the computing task; identifying the database query statement in the code, executing the database query statement, and obtaining result data; obtaining the estimated load of the computing task according to the result data; reading the node load of the distributed computing node, the node load includes the current estimated waiting time in the queue, memory space occupancy and CPU resource occupancy; according to the estimated load and the node load, obtaining all the distributed computing nodes adapted for each computing task, and the estimated completion time of the computing task; obtaining the designated distributed computing node of each computing task according to the estimated completion time, so that the time difference of the estimated completion time of the distributed computing node to complete all the designated computing tasks is minimized.

Description

Technical Field

[0001] Multiple embodiments of this specification relate to the field of information technology, and specifically to a distributed computing load balancing method and system. Background Art

[0002] Distributed computing refers to the process of breaking down computing tasks and executing them in parallel on multiple computers (nodes). These nodes are connected to each other through a network and work together to complete one or more complex tasks. Distributed computing is characterized by parallel processing, resource sharing, strong fault tolerance, high scalability, and heterogeneity. Load balancing is the distribution of workloads to different nodes in a distributed computing environment to ensure that no node is overloaded while other nodes are idle. Load balancing is essential to maintaining system stability and responsiveness. The goal of load balancing technology is to optimize resource usage, maximize throughput, minimize response time, and avoid overloading any node. Common load balancing strategies include Round Robin, Least Connections, Weighted, Geolocation, and Content-Based. Content-aware forwards requests to specific nodes based on the content type or specific parameters of the request. Load balancing is very common in modern Internet applications and services, especially in scenarios that require high availability and scalability, such as e-commerce websites, social media platforms, and online games. However, when the current load balancing technology needs to process the database query results one by one, the number of rows in the database query results is uncertain, resulting in unpredictable specific calculation amount, which may cause node overload. Therefore, it is necessary to improve the load balancing technology. Summary of the invention

[0003] Multiple embodiments of this specification describe a distributed computing load balancing method and system.

[0004] In a first aspect, an embodiment of this specification provides a distributed computing load balancing method, comprising the steps of:

[0005] Receiving a plurality of computing tasks that can be executed in parallel divided from the target task, and reading the codes of the computing tasks;

[0006] Identify a database query statement in the code, execute the database query statement, and obtain result data;

[0007] Obtaining an estimated load of the computing task according to the result data, the estimated load including an estimated time consumption, an estimated memory space occupancy, and an estimated CPU resource occupancy;

[0008] Read the node load of the distributed computing node, where the node load includes the current estimated waiting time in the queue, memory space usage, and CPU resource usage;

[0009] According to the estimated load and the node load, obtain all distributed computing nodes adapted for each computing task and an estimated completion time of the computing task;

[0010] The designated distributed computing node for each computing task is obtained according to the estimated completion time, so that the time difference of the estimated completion time of the distributed computing node to complete all the designated computing tasks is minimized.

[0011] In a second aspect, the embodiments of this specification provide a distributed computing load balancing system, including:

[0012] A receiving module receives a plurality of computing tasks that can be executed in parallel divided from a target task, and reads the codes of the computing tasks;

[0013] A query module, which identifies a database query statement in the code, executes the database query statement, and obtains result data;

[0014] An estimation module, which obtains an estimated load of the computing task according to the result data, wherein the estimated load includes an estimated time consumption, an estimated memory space occupancy, and an estimated CPU resource occupancy;

[0015] A reading module reads the node load of the distributed computing node, wherein the node load includes the current estimated waiting time in the queue, the memory space occupied, and the CPU resource occupied;

[0016] An adaptation module, which obtains all distributed computing nodes adapted for each computing task and an estimated completion time of the computing task according to the estimated load and the node load;

[0017] The designated module obtains the designated distributed computing node of each computing task according to the estimated completion time, so as to minimize the time difference of the estimated completion time of all computing tasks corresponding to the target task.

[0018] In a third aspect, an embodiment of this specification provides an electronic device, including a processor and a memory;

[0019] The processor is connected to the memory;

[0020] The memory is used to store executable program code;

[0021] The processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to execute the method described in any one of the above aspects.

[0022] In a fourth aspect, an embodiment of the present specification provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in any of the above aspects.

[0023] In a fifth aspect, an embodiment of this specification provides a computer program product, including a computer program, which implements the method described in any of the above aspects when executed by a processor.

[0024] The beneficial effects brought by the technical solutions provided by some embodiments of this specification include at least:

[0025] In multiple embodiments of the present specification, the provided distributed computing load balancing method pre-executes the database query statements in the computing task on the designated server, so that the estimated load of the computing task can be relatively more accurately estimated based on the number of result data, thereby helping to schedule the distributed computing load more evenly. By reading the node load of the distributed computing node and combining the estimated load of the computing task, the time difference of the estimated completion time of the distributed computing node to complete all the designated computing tasks is minimized, so that the computing tasks can be integrated faster, the results of the target tasks can be obtained, and the efficiency of distributed computing can be improved. The database query statement is pre-executed on the designated server, thereby avoiding the situation where the distributed node frequently establishes a connection with the server, reducing the pressure on the database, and helping to improve the execution efficiency of the database.

[0026] Other features and advantages of the various embodiments of the present specification will be further disclosed in the following detailed description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of this specification, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0028] Figure 1 A schematic diagram of an application scenario of the distributed computing load balancing method provided in the embodiments of this specification.

[0029] Figure 2 A schematic diagram of the distributed computing load balancing method provided in the embodiments of this specification.

[0030] Figure 3 A flowchart of a method for fusing database query statements provided in an embodiment of this specification.

[0031] Figure 4 A flowchart of a query condition fusion method provided in an embodiment of this specification.

[0032] Figure 5 A schematic diagram of the flow chart of the method for fusing numerical conditions provided in the embodiments of this specification.

[0033] Figure 6 A schematic diagram of a method for estimating time consumption provided in an embodiment of this specification.

[0034] Figure 7 A schematic diagram of a method for estimating memory space occupancy provided in an embodiment of this specification.

[0035] Figure 8 A flowchart of a method for estimating CPU resource usage provided in an embodiment of this specification.

[0036] Fig. 9 A schematic diagram of a distributed computing load balancing system provided in an embodiment of this specification.

[0037] Fig.10 A schematic diagram of an electronic device provided in an embodiment of this specification. DETAILED DESCRIPTION

[0038] The following is an explanation and description of the technical solutions of the embodiments of this specification in conjunction with the drawings of the embodiments of this specification, but the following embodiments are only preferred embodiments of this specification, not all. Based on the embodiments in the implementation mode, other embodiments obtained by those skilled in the art without creative work are all within the scope of protection of this specification.

[0039] The terms "first", "second", "third", etc. in the description and claims of this specification and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0040] In the following description, terms such as "inside", "outside", "up", "down", "left", "right", etc. that indicate directions or positional relationships are only used to facilitate the description of the embodiments and simplify the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore should not be understood as a limitation of this specification.

[0041] The data involved in this manual are all information and data authorized by the user or fully authorized by all parties, and the collection of relevant data complies with the relevant laws, regulations and standards of relevant countries and regions.

[0042] Before introducing the technical solutions provided in this specification, the relevant technologies and application scenarios are introduced.

[0043] With the popularization of the Internet and the development of information technology, a large amount of data has been generated. The traditional centralized processing method can no longer meet the needs of processing this data. With the intensification of corporate competition and the enrichment of services, enterprises need a technology that can dynamically expand computing resources according to demand and flexibly respond to different computing tasks when facing the growing amount of data and complex computing tasks. For example, artificial intelligence, big data analysis, etc., these applications require the system to have strong computing capabilities.

[0044] Distributed computing systems can dynamically adjust computing power according to actual needs by dynamically expanding computing resources, avoiding wasting resources during off-peak hours. For example, during promotions, holidays, or other special events, companies may face sudden surges in traffic or computing needs. Dynamic expansion capabilities can ensure that the system can operate normally during peak periods of promotions and holidays. At the same time, distributed computing systems also have automatic fault recovery capabilities. When the system fails or requires maintenance, dynamic expansion can also ensure that services are not interrupted, thereby improving business continuity and reliability. These advantages have led to the rapid development of distributed computing technology and its widespread application in the field.

[0045] Distributed computing is mainly implemented through cloud computing, containerization and microservices, automation tools, hybrid cloud and multi-cloud technologies. Cloud computing providers (such as AWS, Azure, Google Cloud, etc.) provide elastic computing services, and users can increase or decrease computing resources at any time according to demand. Through containerization (such as Docker) and microservice architecture, applications can be quickly deployed and expanded to improve resource utilization efficiency. Use automation tools (such as Kubernetes, Ansible, etc.) to manage and schedule computing resources to achieve automatic expansion and contraction. Combine the advantages of public and private clouds, or use multiple public cloud service providers to flexibly allocate resources according to actual needs.

[0046] The implementation of distributed computing requires solving data consistency, fault recovery and fault tolerance, load balancing, data security, network management, data management, etc. Data consistency ensures that the data of each node in the distributed system is consistent, especially when performing read and write operations. Fault recovery and fault tolerance require timely detection of faulty nodes and isolation and recovery. Data in distributed computing uses encryption to ensure the security of data transmission and storage. Network management needs to deal with the high latency problem between nodes in different geographical locations. Data management divides large tables into small blocks and stores them on different nodes and performs data migration transparent to the application. The goal of load balancing is to make the load of each node as even as possible, avoiding the situation where some nodes are overloaded while other nodes are idle, so as to maximize resource utilization, reduce task processing time, and improve system reliability and response speed. Ensure that computing resources are reasonably allocated to each computing task, thereby improving the overall performance and efficiency of the system.

[0047] Common load balancing technologies include round-robin, least connections, weighted round-robin, weighted least connections, content-based allocation, etc. Among them, content-based allocation technology allocates requests to the most suitable nodes according to the content of the request or specific parameters. It can optimize the allocation for different types of tasks.

[0048] The distributed computing load balancing method provided in this specification belongs to the category of load balancing technology based on content distribution. Since this specification involves some professional terms, these professional terms will be introduced first below.

[0049] Intensive calculation statements

[0050] Intensive computing statements refer to those codes or code segments that involve a large number of computing operations in the program. They are often encountered in scientific computing, engineering simulation, big data processing, machine learning and other fields. Such statements are characterized by high computational complexity, high computing resource usage, and may require parallel processing or distributed computing to improve efficiency. The following are some typical examples of intensive computing statements. In numerical computing, tasks such as matrix operations and solving linear equations are often required. For example, using Python's NumPy library for matrix multiplication, an exemplary code can be:

[0051] import numpy as np

[0052] # Create two random matrices

[0053] A = np.random.rand(1000, 1000)

[0054] B = np.random.rand(1000, 1000)

[0055] # Compute matrix multiplication

[0056] C = np.dot(A, B)

[0057] This code creates two 1000x1000 random matrices and performs matrix multiplication. Matrix multiplication, np.dot(A, B), is a typical intensive computing task, especially in the case of large matrices.

[0058] Additionally, nested loops are another common form of intensive computation, especially when dealing with multidimensional arrays.

[0059] Gradient calculation in machine learning is another common intensive calculation. In machine learning, especially in deep learning, the gradient descent algorithm needs to calculate the gradient of the loss function with respect to the model parameters. This is an intensive calculation process, especially when the model is complex and the amount of data is large.

[0060] These tasks usually involve a large number of mathematical operations, such as addition, multiplication, matrix operations, etc. When processing large amounts of data, a large amount of memory resources may be consumed. Many intensive computing tasks can be accelerated through parallel processing, such as using acceleration hardware such as GPUs and TPUs. In order to improve efficiency, it is usually necessary to optimize the algorithm, such as using more efficient matrix operation libraries, parallel computing frameworks, etc.

[0061] load

[0062] In distributed computing, "load" usually refers to the workload or task volume undertaken by each node in the system. The "load" here can be understood as the degree of occupation of system resources, including but not limited to CPU usage, memory usage, disk I / O operations, and network traffic. Load is an important indicator for measuring the performance and resource utilization of distributed systems. In order to ensure the efficient operation of distributed systems, load management needs to be carried out effectively. It mainly includes load balancing, dynamic scheduling, resource reservation and allocation, elastic scaling, monitoring and early warning. Load balancing uses load balancing technology to reasonably allocate tasks to each node to ensure that the load of each node is relatively balanced. Dynamic scheduling dynamically adjusts the allocation of tasks according to the real-time load situation, such as allocating new tasks to nodes with lower current load. Resource reservation and allocation reserves resources for key tasks in advance to ensure that important tasks can get enough computing resources. Elastic scaling automatically adjusts the scale of the system according to changes in load, such as adding nodes during high-load periods and reducing nodes during low-load periods. Monitoring and early warning monitors the load of each node in real time, and issues an early warning when the load exceeds the threshold, and takes timely measures.

[0063] Node load

[0064] In a distributed computing environment, node load refers to the resource usage of each computing node when executing a task. Load management is the key to ensuring the efficient operation of a distributed system. Its indicators include but are not limited to CPU usage, memory usage, disk I / O operations, and network traffic. When a distributed computing task 30 includes a database SQL query statement, the SQL query itself may cause a certain load on the database 40, especially when there are many query conditions and data needs to be obtained from multiple tables.

[0065] Probability of probability distribution

[0066] It represents an approximate estimate of the distribution probability, that is, a probability distribution that contains errors or uncertainties to a certain extent. Because the cost of the distribution probability of the values ​​of the statistical column is high, a large number of database query operations are required, especially for database systems with more columns, and its execution efficiency is very low. And the technical solution provided in this specification does not require the accuracy of the distribution probability. Therefore, an approximate estimate of the distribution probability can be used. The probable distribution probability can be obtained by rough observation of the data or based on a limited sample, without the need for an accurate distribution based on a large amount of detailed data. The probable distribution probability contains a certain degree of subjective judgment or prediction based on the model, so it has a certain degree of uncertainty. When the data quality is not high or there are errors in the data collection process, the obtained probable distribution probability will have a certain degree of ambiguity. However, the errors and ambiguity of the probability distribution basically do not affect the implementation of the technical solution provided in this specification.

[0067] Taking the "order shipment confirmation" of the merchant 10 backstage during the Double Eleven e-commerce promotion as an example, which is an application scenario that can best reflect the advantages of the distributed computing load balancing method provided by this specification, the technical solution of this specification is introduced. Each merchant 10 needs to perform the task of order shipment confirmation. Obviously, it is necessary for the merchant 10 to use distributed computing technology to realize order processing during the Double Eleven e-commerce promotion, so load balancing technology is needed. When distributed computing technology is used to realize order processing, it is more appropriate to use content-based distribution-based load balancing technology. The reason is that the distributed computing tasks 30 of "order shipment confirmation" initiated by different merchants 10 have different corresponding data processing volumes, and even huge differences.

[0068] The distributed computing task 30 initiated by the merchant 10 is determined by its order volume. When the merchant 10 has a large order volume, the corresponding distributed computing task 30 has a large amount of computing and takes longer time. However, the problem is that the distributed computing task 30 of "order shipment confirmation" requires querying the database 40 to obtain the corresponding computing amount. Another problem is that the merchant 10 may not include all pending orders into this distributed computing task 30 when initiating the distributed computing task 30 of "order shipment confirmation".

[0069] Exemplarily, the merchant 10 may only want to include the pending orders of product A into this distributed computing task 30, so as to process the shipment of product A. In another exemplary embodiment, the merchant 10 may only include the pending orders that were completed yesterday morning into this distributed computing task 30. In another exemplary embodiment, the merchant 10 may include the pending orders whose delivery addresses are located in several selected provinces into this distributed computing task 30. Or, in combination with the aforementioned screening conditions, such as screening based on product and delivery address conditions, orders that meet the screening conditions will be included in this distributed computing task 30. This results in the method of establishing an index in advance not being able to solve the problem of difficulty in confirming the computational amount of the distributed computing task 30 of "order shipment confirmation".

[0070] If the database 40 is queried first to obtain the number of orders to be shipped by the merchant 10 who initiated the distributed computing task 30, a more accurate confirmation of the computing amount of the distributed computing task 30 can be obtained, but it will lead to an increase in the number of database queries, increase the load on the database system, and affect the efficiency of the distributed computing system.

[0071] To this end, this specification provides a distributed computing load balancing method. Figure 1 , which is a schematic diagram of the application scenario framework of the distributed computing load balancing method provided in this specification. The distributed computing load balancing method provided in this specification is applied on a server, which can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms. In short, it is able to interact with the distributed computing system that ultimately executes the distributed computing task 30 and control the server to which the computing task 30 is assigned.

[0072] First, the merchant 10 initiates a distributed computing task 30, namely, a target task, through the console 20. The console 20 is provided by a distributed computing system, and the technology for dividing the target task into multiple computing tasks 30 that can be executed in parallel can be implemented using the technology that has been disclosed in the art. Then, the server running the distributed computing load balancing method provided in this specification receives the multiple computing tasks that can be executed in parallel divided from the target task, establishes a connection with the database 40, and executes a database query statement. The estimated load 31 of the computing task is obtained based on the result data, and the current node load of the distributed computing node 50 is combined to complete the allocation of the computing task, and the load balancing of the distributed computing node 50 is achieved by allocation.

[0073] The database 40 is a collection of data that is stored in a certain manner, can be shared with multiple accounts, has as little redundancy as possible, and is independent of the application program. A database management system (DBMS) is a computer software system designed for managing the database 40, and generally has basic functions such as storage, interception, security, and backup. Database management systems can be classified according to the database model it supports, such as relational, Extensible Markup Language (XML); or according to the type of computer supported, such as server clusters, mobile phones; or according to the query language used, such as Structured Query Language (SQL), XQuery; or according to performance impulse focus, such as maximum scale, maximum operating speed; or other classification methods. Regardless of the classification method used, some DBMS can cross categories, for example, supporting multiple query languages ​​at the same time.

[0074] Please see attached Figure 2 , a distributed computing load balancing method provided in this specification includes the following steps:

[0075] Step S01) receiving a plurality of computing tasks that can be executed in parallel divided from a target task, and reading the codes of the computing tasks.

[0076] Dividing the target task into multiple computing tasks that can be executed in parallel is an important step in distributed computing. This process is called task decomposition or task partitioning in this field. It includes data parallelism, task parallelism and other methods. When using data parallelism, you can choose to divide the data by uniform partitioning, hash partitioning, range partitioning, random partitioning and other methods. When using task parallelism, divide the task according to its logical steps or functional modules.

[0077] This embodiment uses data parallelism as an example to divide tasks. Exemplarily, the merchant 10 creates a pre-delivery review process for the pending orders that were completed yesterday through the console 20, so that the pending orders that have passed the review can be synchronized to the warehouse for distribution and delivery. As an example, the review process includes address checking (such as whether the address is complete, whether it is a non-existent address, whether the consignee's contact information is formally correct), inventory checking (checking whether the goods involved in the order are in stock in the warehouse, and locking and deducting the corresponding inventory when the order review process passes), and order status checking (checking whether the order status is a pending delivery status, rather than a refund application status). The console 20 divides the target task of pre-delivery review processing for the pending orders that were completed yesterday into multiple computing tasks that can be executed in parallel as follows. The target task is divided into multiple computing tasks according to the product types, delivery address provinces, and order amounts contained in the order.

[0078] Specifically, take product types such as only product A, only product B, only product C, and multiple product types as examples, take province A, province B, province C, and other provinces as examples of the delivery address, and take the amount range (0, 100], (100, 1000], (1000, 20000) as examples of the order amount. Then, 4×4×3, or 48 computing tasks, can be obtained. For example, one computing task a processes the pre-shipment review processing of pending orders of multiple product types, whose delivery addresses are in province A, and whose order amounts are in the range (100, 1000].

[0079] Computational task b processes the pre-shipment review of pending orders for product A only, whose delivery addresses are in other provinces, and whose order amounts are in the range of (0,100]. Computational task c processes the pre-shipment review of pending orders for product A only, whose delivery addresses are in other provinces, and whose order amounts are in the range of (100,1000].

[0080] Computational task d processes the pre-shipment review of pending orders for product B, whose delivery address is province A, and whose order amount is in the interval (0,100]. Computational task e processes the pre-shipment review of pending orders for product B, whose delivery address is province A, and whose order amount is in the interval (100,1000]. After completing the division of the computational tasks and reading the code of the computational tasks, proceed to the next step.

[0081] Step S02) identifying a database query statement in the code, executing the database query statement, and obtaining result data.

[0082] Exemplarily, the main part of the code for computing task b is as follows:

[0083] #Establish a connection with the database

[0084] def connect_to_database(db_path: str) -> sqlite3.Connection:

[0085] conn = sqlite3.connect(db_path)

[0086] return conn

[0087] def process_orders(conn: sqlite3.Connection):

[0088] cursor = conn.cursor()

[0089] # Query orders that meet the conditions

[0090] cursor.execute("""SELECT o.id, o.customer_name, o.shipping_address, o.phone, o.order_amount, oi.product_id, oi.quantity FROM orders o JOIN order_items oi ON o.id = oi.order_id WHERE o.product_id = 'get_product_id(Product A)' AND o.shipping_province != 'Province A / B / C' AND o.order_status = 'Pending Shipment' AND o.order_amount BETWEEN 0 AND 100 """)

[0091] orders = cursor.fetchall()

[0092] for order in orders:

[0093] order_id, customer_name, shipping_address, phone, order_amount, product_id, quantity = order

[0094] # Check address integrity

[0095] if not is_address_complete(shipping_address):

[0096] print(f"Order {order_id}: Address is incomplete.")

[0097] continue

[0098] # Check the address correctness

[0099] if not is_address_correct(shipping_address):

[0100] print(f"Order {order_id}: Address is incorrect.")

[0101] continue

[0102] # Check the phone number is correct

[0103] if not is_phone_number_valid(phone):

[0104] print(f"Order {order_id}: Phone number is invalid.")

[0105] continue

[0106] # Check Inventory

[0107] if not check_product_inventory(product_id, quantity, cursor):

[0108] print(f"Order {order_id}: Product {product_id} does not haveenough inventory.")

[0109] continue

[0110] #If all checks pass, update the order status

[0111] cursor.execute("UPDATE orders SET order_status='Audited' WHERE id=?",(order_id,))

[0112] conn.commit()

[0113] if __name__ == "__main__":

[0114] db_path = "path / to / your / database.db"

[0115] conn = connect_to_database(db_path)

[0116] process_orders(conn)

[0117] conn.close()

[0118] In addition, a special library needs to be set up to implement the audit process. As an example, the library function that checks whether the address is complete and correct is as follows:

[0119] # Check if the address is complete

[0120] def is_address_complete(address: str) ->bool:

[0121] return all(part.strip() for part in address.split(','))

[0122] # Check if the address is correct

[0123] def is_address_correct(address: str) ->bool:

[0124] return true

[0125] Among them, the functions is_address_complete() and is_address_correct() are intensive calculation statements, because these two statements need to process the operation of comparing the address with a large number of address audit rules. The specific implementation does not belong to the technical improvement of the solution provided in this specification, and will not be discussed here. It can be carried out using the technology disclosed in the field.

[0126] The codes of other computing tasks are similar to computing task B. In the above code, cursor.execute() statement and orders = cursor.fetchall() statement are database query statements. Before scheduling computing task B to be executed on distributed computing node 50, the database query statement is executed first to obtain result data.

[0127] Step S03) Obtaining the estimated load 31 of the computing task according to the result data, wherein the estimated load 31 includes the estimated time consumption, the estimated memory space occupancy and the estimated CPU resource occupancy.

[0128] Assume that after the database query statement of computing task b is executed, the number of result rows obtained is 12,800 data. Assume that the size of each order record is about 1KB, the processing time of each order is t = 0.1 second (i.e. 100 milliseconds), and the average number of concurrent tasks of distributed computing nodes 50 is 10.

[0129] Concurrency control involves task scheduling, lock management, etc., which will consume a certain amount of time. Assume that the cost of concurrency control is 10% of each processing, that is, 128s×10%=12.8s.

[0130] The estimated time required to calculate task b is T=12800×0.1÷10+12.8=140.8 seconds.

[0131] The result data has 12,800 records, with a total size of 12,800 × 1KB / record = 12,800KB = 12.5MB. In the process of processing orders, you may need to create some intermediate data structures, such as those used to store address verification results, phone number verification results, etc. The size of these data structures depends on the specific implementation details, but they are usually not very large. Assuming that each order requires an additional 100 bytes to store intermediate data, a total of 12,800 × 100 bytes / record = 1,280,000 bytes = 1.25MB is required. Each database connection and transaction processing will also occupy a certain amount of memory resources. Assuming that each connection occupies 1MB of memory, 10 concurrent connections require a total of: 10 connections × 1MB / connection = 10MB.

[0132] The estimated memory space occupied by computing task b is 12.5MB (query results) + 1.25MB (intermediate data structure) + 10MB (database connection) = 23.75MB.

[0133] This manual pre-executes the database query statement and directly gives the result data, so there is no need to execute the database query in the processing of each order. The processing logic of each order includes address verification, phone number verification, inventory check, etc. Assume that each order processing requires 10% of CPU resources. Concurrency control involves task scheduling, lock management, etc., which may have a certain CPU overhead. Assume that the overhead of concurrency control is 10% of each processing. Then the estimated CPU resource occupancy of calculation task b is 11% of CPU resources.

[0134] In this embodiment, the estimation of the estimated load 31 of the computing task is relatively rough, and a more accurate scheme for estimating the estimated load 31 is disclosed in subsequent embodiments.

[0135] Step S04) Read the node load of the distributed computing node 50, wherein the node load includes the current estimated waiting time in the queue, memory space occupancy, and CPU resource occupancy.

[0136] Obtain the queue list of the current computing tasks of the distributed computing node 50, and calculate the sum of the estimated time consumption of the computing tasks in the queue list. Obtain the estimated time consumption of the computing task currently being executed by the distributed computing node 50, and obtain the duration of the currently executed computing task based on the time difference between the time when the currently executed computing task is scheduled into the CPU and the current time. Then obtain the estimated duration that the currently executed computing task still needs to be executed, and add the sum of the estimated time consumption of the computing tasks in the queue list to obtain the current estimated waiting duration of the distributed computing node 50. The memory space occupancy and CPU resource occupancy can be directly obtained from the resource manager of the distributed computing node 50. Or the distributed node actively reports the memory space occupancy and CPU resource occupancy to the designated server at a preset period.

[0137] Step S05) According to the estimated load 31 and the node load, all the distributed computing nodes 50 adapted for each of the computing tasks and the estimated completion time of the computing tasks are obtained.

[0138] Obtaining all the distributed computing nodes 50 adapted for each of the computing tasks specifically refers to that after the distributed computing node 50 has waited in the current queue for the estimated time, the free memory space of the distributed computing node 50 is greater than the estimated memory space occupied by the computing task b. Or at the current moment, the free memory space of the distributed computing node 50 is greater than the estimated memory space occupied by the computing task b. The CPU resource occupancy of the distributed computing node 50 can be released within a preset time threshold after the current queue for the estimated time.

[0139] The estimated completion time of task b is calculated to be equal to the current time, plus the current estimated waiting time in the queue, plus the estimated time taken.

[0140] Step S06) Obtain the designated distributed computing node 50 for each computing task according to the estimated completion time, so that the time difference of the estimated completion time of the distributed computing node 50 to complete all designated computing tasks is minimized.

[0141] Table 1 Estimated load of computing tasks

[0142]

[0143] Table 2 Node load of distributed computing nodes

[0144]

[0145] According to Table 1 and Table 2, it can be seen that the memory space and CPU resource occupancy of the distributed computing node 50 are sufficient, so nodes 1 to 3 are adapted to each computing task. The final allocation result is Node 1: Task b, Task f, Task i, Node 2: Task c, Task g, Node 3: Task d, Task e, Task h. The estimated completion time for node 1 to complete all the designated computing tasks is 847.8 s, the estimated completion time for node 2 to complete all the designated computing tasks is 762 s, and the estimated completion time for node 3 to complete all the designated computing tasks is 745 s. The time difference between node 3 and node 1 is 102.8 s, which is the case where the time difference of the estimated completion time of the distributed computing node 50 to complete all the designated computing tasks is minimized.

[0146] The new technical effects that can be achieved by this embodiment include: pre-executing the database query statement in the computing task on the designated server, so that the estimated load 31 of the computing task can be estimated more accurately based on the number of result data, thereby helping to schedule the distributed computing load more evenly. By reading the node load of the distributed computing node 50 and combining the estimated load 31 of the computing task, the time difference of the estimated completion time of the distributed computing node 50 to complete all the designated computing tasks is minimized, so that the computing tasks can be integrated faster, the results of the target tasks can be obtained, and the efficiency of distributed computing can be improved. Pre-executing the database query statement on the designated server avoids the situation where the distributed node frequently establishes a connection with the server, reduces the pressure on the database, and helps to improve the execution efficiency of the database.

[0147] Example 2

[0148] Please see attached Figure 3 In this embodiment, the method of identifying the database query statement in the code, executing the database query statement, and obtaining result data includes the steps of:

[0149] Step S201) Read the codes of multiple computing tasks at one time and extract all database query statements in the codes.

[0150] Exemplarily, the codes of computing task b and computing task c are read at one time, and the database query statements for computing task b and computing task c are extracted as follows:

[0151] cursor.execute("""SELECT o.id, o.customer_name, o.shipping_address,o.phone, o.order_amount, oi.product_id, oi.quantity FROM orders o JOIN order_items oi ON o.id = oi.order_id WHERE o.product_id = 'get_product_id(Product A)'AND o.shipping_province != 'Province A / B / C' AND o.order_status = 'To be shipped' ANDo.order_amount BETWEEN 0 AND 100 """)

[0152] orders = cursor.fetchall()

[0153] and

[0154] cursor.execute("""SELECT o.id, o.customer_name, o.shipping_address,o.phone, o.order_amount, oi.product_id, oi.quantity FROM orders o JOIN order_items oi ON o.id = oi.order_id WHERE o.product_id = 'get_product_id(Product A)'AND o.shipping_province != 'Province A / B / C' AND o.order_status = 'To be shipped' ANDo.order_amount BETWEEN 100 AND 1000 """)

[0155] orders = cursor.fetchall()

[0156] After extracting all database query statements in the code, proceed to the next step.

[0157] Step S202) Database query statements for the same data table are identified and grouped together, and query conditions of the database query statements in the same group are merged, so that the database query statements in the same group are merged, which is called a merged query statement.

[0158] The database query statements of computing tasks b and c are for the same data table and are grouped together. The query conditions of the database query statements of computing tasks b and c are merged to merge the two database query statements, which is called a fused query statement. It can be executed in one query, reducing the execution of one database query. Although the total number of result data remains unchanged, the fused query statement only requires the database to perform data screening once, while the original two database query statements require the database to perform data screening twice. Obviously, it can improve the query efficiency of the database and reduce the workload of the database.

[0159] Exemplarily, the fusion query statement is:

[0160] cursor.execute("""SELECT o.id, o.customer_name, o.shipping_address,o.phone, o.order_amount, oi.product_id, oi.quantity FROM orders o JOIN order_items oi ON o.id = oi.order_id WHERE o.product_id = 'get_product_id(Product A)'AND o.shipping_province != 'Province A / B / C' AND o.order_status = 'To be shipped' ANDo.order_amount BETWEEN 0 AND 1000 """)

[0161] orders = cursor.fetchall()

[0162] That is, the numerical ranges (0, 100] and (100, 1000] are merged into (0, 1000]. The WHERE statement as a whole constitutes the query condition referred to in this specification, and o.order_amount BETWEEN 0 AND 1000 constitutes a column condition. After completing the division of the computing tasks and reading the code of the computing tasks, proceed to the next step.

[0163] Step S203) Connect to the database, execute the fusion query statement, and obtain fusion query result data.

[0164] Execute the fusion query statement to obtain all pending orders for product A with delivery addresses in other provinces and order amounts in the range (0,1000] as the fusion query result data. Add the assignment of the fusion query result data to an array variable in the programs of calculation tasks b and c, and then add a filter judgment statement in the loop body that traverses each element of the array variable.

[0165] For calculation task b, if the order amount is not in (0,100], the execution of this loop body is skipped. That is:

[0166] if order_amount >= 100:

[0167] continue

[0168] else:

[0169] #Subsequent check statements

[0170] The continue statement can be used to directly skip the processing of this order data and process the next order data. For computing task b, if the order amount is not in (100,1000], the execution of this loop body is skipped. Thus, only one database query is required to achieve two computing tasks. The reduction in the efficiency of the computing task is far less than the improvement in efficiency by saving one database query, so the efficiency of distributed computing can be improved as a whole.

[0171] Please see attached Figure 4 The method for fusing query conditions of database query statements in the same group comprises the steps of:

[0172] Step S301) When the conditions of multiple columns in the query conditions are different, multiple database query statements are executed in one database connection. When the condition of only one column in the query conditions is different, the numerical range condition and the numerical value condition of the same column in the query conditions are combined to obtain a combined numerical condition by fusing the numerical range condition and the numerical value condition.

[0173] For example, when only the condition of o.order_amount is different in the database query statement, the numerical range condition is merged by merging o.order_amount BETWEEN 0 AND 100 and o.order_amount BETWEEN 100 AND1000 into o.order_amount BETWEEN 0 AND 1000. After the result data reference code, add the corresponding filter statement.

[0174] The database query statements of computing tasks b and d belong to the situation when the conditions of multiple columns in the query conditions are different. At this time, multiple database query statements can be executed in one database connection, which can reduce the number of database connections and reduce the database pressure to a certain extent.

[0175] In another example, when only the condition of o.product_id is different in the database query statement, the fusion of the numerical value conditions is to merge o.product_id='3012' and o.product_id='4504' into o.product_id='3012' OR o.product_id='4504'.

[0176] Step S302) Perform a union operation on the label value conditions for the same column in the query conditions to obtain a fused label condition.

[0177] As another example, when only the condition of o.shipping_province is different in the database query statement, the label value conditions o.shipping_province != 'province A / B / C' and o.shipping_province = 'province A' are merged into o.shipping_province != 'province B / C'.

[0178] Step S303) Obtain the possible value range of the column corresponding to the fused numerical condition, and when the ratio of the numerical range corresponding to the fused numerical condition to the possible value range exceeds a preset ratio threshold, delete the fused numerical condition.

[0179] For example, when o.order_amount BETWEEN 0 AND 100 and o.order_amount BETWEEN100 AND 1000 are merged into o.order_amount BETWEEN 0 AND 1000, the merged value range is (0,1000], and the proportion of the possible value range (0,20000] does not exceed the preset proportion threshold, and no processing is performed.

[0180] In another example, when o.order_amount BETWEEN 100 AND 1000 and o.order_amount>1000, they are merged into o.order_amount>100. The ratio of the merged value range (100,20000] to the possible value range (0,20000] exceeds the preset ratio threshold. At this time, o.order_amount can no longer be filtered. That is, the condition setting for the column o.order_amount in the database query statement is directly deleted.

[0181] On the other hand, obtain the possible value label set of the column corresponding to the fusion label condition, and when the fusion label condition covers the possible value label set, delete the fusion label condition. For example, o.shipping_province = 'Province A', o.shipping_province = 'Province B', o.shipping_province = 'Province C' and o.shipping_province != 'Province A / B / C', since all possible values ​​of o.shipping_province are covered, the fused query statement does not need to filter o.shipping_province, that is, directly delete the condition setting for column o.shipping_province.

[0182] On the other hand, before deleting the fusion value condition, please refer to the attached Figure 5 , perform the following steps:

[0183] Step S401) Read a preset checklist, which records the probabilities of the probable distribution of columns.

[0184] Step S402) Obtain the probable distribution probability of the column corresponding to the fusion numerical condition according to the lookup table.

[0185] Step S403) When the probability of the probable distribution corresponding to the numerical range corresponding to the fused data condition is lower than a preset threshold, the fused numerical condition is prevented from being deleted; when the probability of the probable distribution corresponding to the numerical range corresponding to the fused data condition is not lower than the preset threshold, the fused numerical condition is deleted.

[0186] Exemplarily, the checklist records that the distribution of column o.shipping_province is uniformly distributed. Before this distributed computing task, based on the historical sales order data of product A, the distribution probability of the order amount o.order_amount when the consumer only purchases product A is obtained, so that the probable distribution probability of column o.order_amount can be recorded in the checklist, assuming it is {'<=100':45%,'>100 AND<=1000':40%,'>1000':15%}. Assuming the preset threshold is 80%, when the fused data condition is o.order_amount>100, since the probable distribution probability corresponding to the numerical range corresponding to the fused data condition is 55%, which is lower than the preset threshold of 80%, the operation of deleting the fused numerical condition is not performed at this time.

[0187] On the other hand, the method for presetting the checklist includes the steps of:

[0188] Preset several distribution probability functions;

[0189] Selecting a distribution probability function from a plurality of the distribution probability functions as the distribution probability function of the column according to the attribute of the column;

[0190] The probable distribution probability of the column is obtained according to the distribution probability function.

[0191] Reading the order data that has been generated by e-commerce and calculating the probable distribution probability still requires computing power. Therefore, you can manually or through a large model, directly based on experience or knowledge learned from the large model, select a distribution probability function from several distribution probability functions as the distribution probability function of the column. Just make a rough estimate. For example, the order amount o.order_amount can be directly set to conform to the Zhengtai distribution, and further calculate each amount interval segment to obtain the probable distribution probability. The rest is the same as Example 1.

[0192] Compared with Embodiment 1, the technical effects newly achieved by this embodiment include:

[0193] By fusing query statements, we can further reduce database connections and the number of database queries, relieve database pressure, reduce the time it takes to wait for a database response in data processing tasks, and improve the execution efficiency of data processing tasks. Deleting fused data that meets the relevant conditions can improve the execution efficiency of the database.

[0194] Example 3

[0195] In this embodiment, please refer to the attached Figure 6 The method for obtaining the estimated time consumption of the computing task according to the result data comprises the steps of:

[0196] Step S501) reading a preset statement estimated time consumption table, and obtaining the estimated time consumption of each statement of the code of the computing task according to the statement estimated time consumption table;

[0197] Step S502) obtaining the execution times of each statement of the code of the computing task according to the number of rows of the fusion query result data;

[0198] Step S503) According to the estimated time consumption and execution times of each statement, the result data is obtained to obtain the estimated time consumption of the computing task.

[0199] For example, the code of data processing task b includes sqlite3.connect statement, conn.cursor statement, SELECT statement, for loop body statement, is_address_complete statement, is_address_correct statement, is_phone_number_valid statement, and check_product_inventory statement. In the statement estimated time table, the estimated time of sqlite3.connect statement, conn.cursor statement, and SELECT statement is recorded as 100ms because the response of the database is required. The for loop body statement is the product of the number of statements in the loop body and 100ms, and then multiplied by the number of loops. The number of statements in the loop body is 6, that is, the product of 600ms and the number of loops.

[0200] Please see attached Figure 7 The method for obtaining the estimated memory space occupied by the computing task according to the result data comprises the steps of:

[0201] Step S601) summing the storage space occupied by the code of the computing task and the storage space occupied by the result data to obtain the total occupied storage space;

[0202] Step S602) multiplying the total occupied storage space by a preset coefficient;

[0203] Step S603) When the product does not exceed a preset upper threshold, the product is used as the estimated memory space occupancy; when the product exceeds the preset upper threshold, the upper threshold is used as the estimated memory space occupancy.

[0204] The storage space occupied by the code refers to the size of the code required to perform the computing task, and the storage space occupied by the result data refers to the sum of the size of the result data obtained from the database query during the execution of the computing task, the generated intermediate data, and the final result data. The preset coefficient is an empirical coefficient used to adjust the total occupied storage space to reflect the actual memory usage. Exemplarily, the coefficient can be set to 1.2.

[0205] The preset upper threshold refers to the maximum memory usage allowed by the distributed computing node 50, which is generally the memory capacity of the distributed computing node 50. When this upper threshold is exceeded, the estimated memory space usage will take the upper threshold. At this time, the distributed computing node 50 will use virtual memory to use part of the hard disk space as memory space.

[0206] Please see attached Figure 8 The method for obtaining the estimated CPU resource occupancy of the computing task according to the result data comprises the steps of:

[0207] Step S701) reading a preset intensive computing statement table, the intensive computing statement table recording the single CPU resource occupancy of intensive computing statements and corresponding estimated statements, and comparing the code of the computing task with the intensive computing statement table;

[0208] Step S702) when the code of the computing task does not contain the statement recorded in the intensive computing statement table, using the preset CPU resource occupancy as the estimated CPU resource occupancy of the computing task;

[0209] Step S703) When the code of the computing task contains an intensive computing statement recorded in the intensive computing statement table, the number of executions of the intensive computing statement is obtained according to the result data, and the estimated CPU resource occupancy of the computing task is obtained according to the number of executions, the parallel configuration and the estimated single CPU resource occupancy of the statement.

[0210] In computing task b, is_address_complete, is_address_correct, is_phone_number_valid, and check_product_inventory are intensive computing statements. These statements need to perform more operations. In the intensive computing statement table, the estimated single CPU resource usage of these statements is recorded. The estimated single CPU resource usage of the statement is obtained by averaging the CPU resource usage obtained by monitoring during multiple historical executions. During historical executions, only intensive computing statements are executed, and the CPU resource usage of intensive computing statements is obtained by setting a comparison execution, that is, in one execution, intensive computing statements are executed together with other statements, and in another execution, only the same other statements are executed. The CPU resource usage of intensive computing statements can be obtained by comparing them twice. Since the difference in CPU resources occupied when the CPU executes ordinary statements has little impact, the preset CPU resource usage can be used directly.

[0211] When there is an intensive computing statement, the number of executions of the intensive computing statement is obtained according to the result data, and the estimated CPU resource occupancy of the computing task is obtained according to the number of executions, the parallel configuration and the estimated statement single CPU resource occupancy. When the parallel configuration is non-parallel, the estimated CPU resource occupancy is equal to the estimated statement single CPU resource occupancy. When the parallel configuration is to automatically generate the number of parallels according to the number of executions, the estimated CPU resource occupancy is equal to the product of the estimated statement single CPU resource occupancy and the number of parallels. Exemplarily, when the number of executions is 100 times, the number of parallels is 3, and the estimated statement single CPU resource occupancy is 5%, then the estimated CPU resource occupancy is equal to 15% (ie 3×5%). Another exemplary example, when the number of executions is 10,000 times, the number of parallels is 10, and the estimated statement single CPU resource occupancy is 5%, then the estimated CPU resource occupancy is equal to 50% (ie 10×5%). The parallel configuration determines that when the distributed computing node 50 faces multiple executions, several parallel threads are configured for processing. The rest is the same as Example 1.

[0212] Compared with Embodiment 1, the technical effects newly achieved by this embodiment include:

[0213] By improving the estimated time consumption, estimated memory space occupancy and estimated CPU resource occupancy, it is possible to estimate the time consumption, memory space occupancy and CPU resource occupancy relatively more accurately, making the scheduling of computing tasks more appropriate, helping to improve the balance of distributed computing loads, and helping to improve the efficiency of distributed computing.

[0214] On the other hand, this specification provides a distributed computing load balancing system, see the attached Fig. 9 ,include:

[0215] A receiving module 100 receives a plurality of computing tasks that can be executed in parallel divided from a target task, and reads the codes of the computing tasks;

[0216] A query module 200, which identifies a database query statement in the code, executes the database query statement, and obtains result data;

[0217] An estimation module 300 obtains an estimated load 31 of the computing task according to the result data, wherein the estimated load 31 includes an estimated time consumption, an estimated memory space occupancy, and an estimated CPU resource occupancy;

[0218] The reading module 400 reads the node load of the distributed computing node 50, wherein the node load includes the current estimated waiting time in the queue, the memory space occupied, and the CPU resource occupied;

[0219] The adaptation module 500 obtains all the distributed computing nodes 50 adapted for each of the computing tasks and the estimated completion time of the computing tasks according to the estimated load 31 and the node load;

[0220] The designation module 600 obtains the designated distributed computing node 50 for each computing task according to the estimated completion time, so as to minimize the time difference of the estimated completion times of all computing tasks corresponding to the target task.

[0221] See also Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of this specification is shown.

[0222] like Fig.10As shown, the electronic device 1100 may include: at least one processor 1101, at least one network interface 1104, a user interface 1103, a memory 1105 and at least one communication bus 1102. Among them, the communication bus 1102 can be used to realize the connection and communication of the above-mentioned components. Among them, the user interface 1103 may include a button, and the optional user interface may also include a standard wired interface and a wireless interface. Among them, the network interface 1104 may include but is not limited to a Bluetooth module, an NFC module, a Wi-Fi module, etc. Among them, the processor 1101 may include one or more processing cores. The processor 1101 uses various interfaces and lines to connect various parts in the entire electronic device 1100, and executes various functions and processes data of the routing device 1100 by running or executing instructions, programs, code sets or instruction sets stored in the memory 1105, and calling data stored in the memory 1105. Optionally, the processor 1101 can be implemented in at least one hardware form of DSP, FPGA, and PLA. The processor 1101 may integrate one or a combination of CPU, GPU and modem, etc. Among them, the CPU mainly processes the operating system, user interface and application programs, etc.; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communication.

[0223] It is understandable that the above-mentioned modem may not be integrated into the processor 1101, but may be implemented by a separate chip.

[0224] Among them, the memory 1105 may include RAM or ROM. Optionally, the memory 1105 includes a non-transitory computer-readable medium. The memory 1105 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 1105 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 1105 may also be at least one storage device located away from the aforementioned processor 1101. The memory 1105 as a computer storage medium may include an operating system, a network communication module, a user interface module and an application. The processor 1101 may be used to call the application stored in the memory 1105 and execute the methods in the above-mentioned multiple embodiments.

[0225] The embodiments of this specification also provide a computer-readable storage medium, which stores instructions, and when the instructions are executed on a computer or a processor, the computer or the processor executes the multiple steps in the above embodiments. If the components of the above electronic device are implemented in the form of software functional units and sold or used as independent products, they can be stored in the computer-readable storage medium.

[0226] The embodiments of this specification also provide a computer program product, including a computer program, which implements multiple steps in the above embodiments when executed by a processor.

[0227] In the absence of conflict, the technical features in this embodiment and implementation scheme can be combined arbitrarily.

[0228] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes a plurality of computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of this specification is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes multiple available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital versatile disc (DVD)), or a semiconductor medium (eg, a solid state drive (SSD)).

[0229] When implemented by hardware or firmware, the aforementioned method flow is programmed into the hardware circuit to obtain the corresponding hardware circuit structure and realize the corresponding function. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit, and its logic function is determined by the user programming the device. The designer programs by himself to "integrate" a digital system on a PLD, without asking a chip manufacturer to design and make a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when writing program development, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL), and HDL is not just one, but many. Those skilled in the art should also be aware that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit to easily obtain the hardware circuit that implements the logic method flow.

[0230] The embodiments described above are merely preferred embodiments of this specification and are not intended to limit the scope of this specification. Without departing from the design spirit of this specification, various modifications and improvements made to the technical solutions of this specification by ordinary technicians in this field should fall within the scope of protection determined by the claims of this specification.

Claims

1. A distributed computing load balancing method, characterized in that: Includes steps: Receiving a plurality of computing tasks that can be executed in parallel divided from the target task, and reading the codes of the computing tasks; Identify a database query statement in the code, execute the database query statement, and obtain result data; Obtaining an estimated load of the computing task according to the result data, the estimated load including an estimated time consumption, an estimated memory space occupancy, and an estimated CPU resource occupancy; Read the node load of the distributed computing node, where the node load includes the current estimated waiting time in the queue, memory space usage, and CPU resource usage; According to the estimated load and the node load, obtain all distributed computing nodes adapted for each computing task and an estimated completion time of the computing task; Obtain the designated distributed computing node for each computing task according to the estimated completion time, so that the time difference of the estimated completion time of all the designated computing tasks by the distributed computing node is minimized; The method of identifying a database query statement in the code, executing the database query statement, and obtaining result data comprises the steps of: Reading the codes of the plurality of computing tasks at one time, and extracting all database query statements in the codes; Identify database query statements for the same data table and group them together, merge the query conditions of the database query statements in the same group, and thus merge the database query statements in the same group, which is called fused query statements; Connecting to the database, executing the fusion query statement, and obtaining fusion query result data; The method for obtaining the estimated time consumption of the computing task according to the result data comprises the steps of: Reading a preset statement estimated time consumption table, and obtaining an estimated time consumption of each statement of the code of the computing task according to the statement estimated time consumption table; Obtaining the execution times of each statement of the code of the computing task according to the number of rows of the fusion query result data; Obtain the result data to obtain the estimated time consumption of the computing task according to the estimated time consumption and execution times of each statement; The method for obtaining the estimated memory space occupied by the computing task according to the result data comprises the steps of: Sum the storage space occupied by the code of the computing task and the storage space occupied by the result data to obtain the total occupied storage space; Multiplying the total occupied storage space by a preset coefficient; When the product does not exceed the preset upper threshold, the product is used as the estimated memory space occupancy; when the product exceeds the preset upper threshold, the upper threshold is used as the estimated memory space occupancy; The method for obtaining the estimated CPU resource occupancy of the computing task according to the result data comprises the steps of: Reading a preset intensive computing statement table, the intensive computing statement table records the single CPU resource occupancy of intensive computing statements and corresponding estimated statements, and comparing the code of the computing task with the intensive computing statement table; When the code of the computing task does not contain the statements recorded in the intensive computing statement table, using the preset CPU resource occupancy as the estimated CPU resource occupancy of the computing task; When the code of the computing task contains intensive computing statements recorded in the intensive computing statement table, the number of executions of the intensive computing statements is obtained according to the result data, and the estimated CPU resource occupancy of the computing task is obtained according to the number of executions, parallel configuration and the estimated single CPU resource occupancy of the statement.

2. A distributed computing load balancing method according to claim 1, characterized in that: The method for fusing query conditions of database query statements in the same group comprises the steps of: When the conditions of multiple columns in the query conditions are different, multiple database query statements are executed in one database connection. When the conditions of only one column in the query conditions are different, the value range condition and the value condition of the same column in the query conditions are combined to obtain a combined value condition by combining the value range condition and the value condition. Perform a union operation on the label value conditions of the same column in the query conditions to obtain the fused label condition; The possible value range of the column corresponding to the fused numerical condition is obtained, and when the ratio of the numerical range corresponding to the fused numerical condition to the possible value range exceeds a preset ratio threshold, the fused numerical condition is deleted.

3. A distributed computing load balancing method according to claim 2, characterized in that: Before deleting the fusion numerical condition, read a preset checklist, the checklist recording the probabilities of the probable distribution of the columns; Obtaining the probable distribution probability of the column corresponding to the fusion numerical condition according to the lookup table; When the probability distribution corresponding to the numerical range corresponding to the fused numerical condition is lower than a preset threshold, the fused numerical condition is prevented from being deleted; when the probability distribution corresponding to the numerical range corresponding to the fused numerical condition is not lower than the preset threshold, the fused numerical condition is deleted; The method for presetting the checklist includes the following steps: Preset several distribution probability functions; Selecting a distribution probability function from a plurality of the distribution probability functions as the distribution probability function of the column according to the attribute of the column; The probable distribution probability of the column is obtained according to the distribution probability function.

4. A distributed computing load balancing system, characterized in that: include: A receiving module receives a plurality of computing tasks that can be executed in parallel divided from a target task, and reads the codes of the computing tasks; A query module, which identifies a database query statement in the code, executes the database query statement, and obtains result data; An estimation module, which obtains an estimated load of the computing task according to the result data, wherein the estimated load includes an estimated time consumption, an estimated memory space occupancy, and an estimated CPU resource occupancy; A reading module reads the node load of the distributed computing node, wherein the node load includes the current estimated waiting time in the queue, the memory space occupied, and the CPU resource occupied; An adaptation module, which obtains all distributed computing nodes adapted for each computing task and an estimated completion time of the computing task according to the estimated load and the node load; A designated module obtains a designated distributed computing node for each computing task according to an estimated completion time, so that the time difference of the estimated completion times of all computing tasks corresponding to the target task is minimized; The method of identifying a database query statement in the code, executing the database query statement, and obtaining result data comprises the steps of: Reading the codes of the plurality of computing tasks at one time, and extracting all database query statements in the codes; Identify database query statements for the same data table and group them together, merge the query conditions of the database query statements in the same group, and thus merge the database query statements in the same group, which is called fused query statements; Connecting to the database, executing the fusion query statement, and obtaining fusion query result data; The method for obtaining the estimated time consumption of the computing task according to the result data comprises the steps of: Reading a preset statement estimated time consumption table, and obtaining an estimated time consumption of each statement of the code of the computing task according to the statement estimated time consumption table; Obtaining the execution times of each statement of the code of the computing task according to the number of rows of the fusion query result data; Obtain the result data to obtain the estimated time consumption of the computing task according to the estimated time consumption and execution times of each statement; The method for obtaining the estimated memory space occupied by the computing task according to the result data comprises the steps of: Sum the storage space occupied by the code of the computing task and the storage space occupied by the result data to obtain the total occupied storage space; Multiplying the total occupied storage space by a preset coefficient; When the product does not exceed the preset upper threshold, the product is used as the estimated memory space occupancy; when the product exceeds the preset upper threshold, the upper threshold is used as the estimated memory space occupancy; The method for obtaining the estimated CPU resource occupancy of the computing task according to the result data comprises the steps of: Reading a preset intensive computing statement table, the intensive computing statement table records the single CPU resource occupancy of intensive computing statements and corresponding estimated statements, and comparing the code of the computing task with the intensive computing statement table; When the code of the computing task does not contain the statements recorded in the intensive computing statement table, using the preset CPU resource occupancy as the estimated CPU resource occupancy of the computing task; When the code of the computing task contains intensive computing statements recorded in the intensive computing statement table, the number of executions of the intensive computing statements is obtained according to the result data, and the estimated CPU resource occupancy of the computing task is obtained according to the number of executions, parallel configuration and the estimated single CPU resource occupancy of the statement.

5. An electronic device, characterized in that including a processor and a memory; The processor is connected to the memory; The memory is used to store executable program code; The processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to execute the method according to any one of claims 1 to 3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

7. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images