A distributed data mining task scheduling method and system

Abstract

The embodiment of this specification discloses a distributed data mining task scheduling method and system. The method includes the following steps: reading multiple data processing tasks that can be executed in parallel divided from a target data mining task; reading the code of the data processing task, identifying the database query statement in the code, and executing the database query statement to obtain result data; converting the database query statement into a result data reference code; obtaining the estimated time consumption of the data processing task according to the result data; and distributing the multiple data processing tasks to multiple nodes for execution according to the estimated time consumption, so as to minimize the time difference of the multiple distributed nodes in completing the distributed data processing tasks.

Description

Technical Field

[0001] Multiple embodiments of this specification relate to distributed computing technology, and specifically to a distributed data mining task scheduling method and system. Background Art

[0002] Distributed computing is a computing technology that connects multiple independent computers (nodes) through a network to work together to complete a task. Nodes can share resources and information, and can work together to improve computing efficiency and processing power. Distributed computing has the advantages of parallel processing, resource sharing, strong fault tolerance, and high scalability. Distributed computing has a wide range of applications, from high-performance computing, big data processing to cloud computing services.

[0003] Distributed data mining is a data analysis method performed in a distributed computing environment. It uses a distributed architecture to process large-scale data sets that may be too large to be effectively analyzed on a single machine. The main features of distributed data mining include distributed data storage, communication overhead between nodes, ensuring data security and privacy when processing sensitive data, and the need to integrate local results on each node into global results. Distributed data mining is usually used to deal with problems with large data sets, such as user behavior analysis, recommendation system construction, market trend prediction and other scenarios. At present, the scheduling of data processing tasks divided from distributed data mining tasks follows the scheduling method of distributed computing, resulting in nodes frequently needing to establish database connections and perform data query operations, which brings a serious burden to the database system and restricts the execution efficiency of distributed data mining tasks. Therefore, it is necessary to improve the scheduling technology of data processing tasks. Summary of the invention

[0004] Multiple embodiments of this specification describe a distributed data mining task scheduling method and system.

[0005] In a first aspect, the embodiments of this specification provide a distributed data mining task scheduling method, comprising the steps of:

[0006] Read multiple data processing tasks that can be executed in parallel divided by the target data mining task;

[0007] Reading the code of the data processing task, identifying the database query statement in the code, and executing the database query statement to obtain result data;

[0008] Converting the database query statement into a result data reference code;

[0009] Obtaining an estimated time consumption of the data processing task according to the result data;

[0010] According to the estimated time consumption, the plurality of data processing tasks are distributed to the plurality of nodes for execution, so that the time difference for the plurality of distributed nodes to complete the distributed data processing tasks is minimized.

[0011] In a second aspect, the embodiments of this specification provide a distributed data mining task scheduling system, including:

[0012] A reading module reads multiple data processing tasks that can be executed in parallel divided by the target data mining task;

[0013] A pre-execution module reads the code of the data processing task, identifies the database query statement in the code, and executes the database query statement to obtain result data;

[0014] A conversion module converts the database query statement into a result data reference code;

[0015] An estimation module, for obtaining an estimated time consumption of the data processing task according to the result data;

[0016] The scheduling module distributes the plurality of data processing tasks to the plurality of nodes for execution according to the estimated time consumption, so as to minimize the time difference for the plurality of distributed nodes to complete the distributed data processing tasks.

[0017] In a third aspect, an embodiment of this specification provides an electronic device, including a processor and a memory;

[0018] The processor is connected to the memory;

[0019] The memory is used to store executable program code;

[0020] The processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to execute the method described in any one of the above aspects.

[0021] In a fourth aspect, an embodiment of the present specification provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in any of the above aspects.

[0022] In a fifth aspect, an embodiment of this specification provides a computer program product, including a computer program, which implements the method described in any of the above aspects when executed by a processor.

[0023] The beneficial effects brought by the technical solutions provided by some embodiments of this specification include at least:

[0024] In multiple embodiments of the present specification, the provided distributed data mining task scheduling method can more accurately obtain the estimated time consumption of data processing tasks, so that the distributed data mining tasks can be better scheduled and the execution efficiency of distributed data mining tasks can be improved; the database query statements can be executed more concentratedly, so that multiple data processing task database query statements can be executed in one database connection, that is, the number of times the database needs to establish connections can be reduced, reducing the database pressure; by fusing query statements, the number of database connections and database queries can be further reduced, the database load pressure can be reduced, the time waiting for database responses in data processing tasks can be reduced, and the execution efficiency of data processing tasks can be improved.

[0025] Other features and advantages of the various embodiments of the present specification will be further disclosed in the following detailed description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of this specification, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 This is a schematic diagram of the application scenario of the distributed data mining task scheduling method provided in this specification.

[0028] Figure 2 This is a flow chart of the distributed data mining task scheduling method provided in this specification.

[0029] Figure 3 This is a flowchart of the conversion result data reference code method provided in this manual.

[0030] Figure 4 A flowchart of a method for obtaining an estimated time consumption for a data processing task provided in this specification.

[0031] Figure 5 This is a schematic diagram of the distributed data mining task scheduling system provided in this manual.

[0032] Figure 6 A schematic diagram of an electronic device provided in an embodiment of this specification. DETAILED DESCRIPTION

[0033] The following is an explanation and description of the technical solutions of the embodiments of this specification in conjunction with the drawings of the embodiments of this specification, but the following embodiments are only preferred embodiments of this specification, not all. Based on the embodiments in the implementation mode, other embodiments obtained by those skilled in the art without creative work are all within the scope of protection of this specification.

[0034] The terms "first", "second", "third", etc. in the description and claims of this specification and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0035] In the following description, terms such as "inside", "outside", "up", "down", "left", "right", etc. that indicate directions or positional relationships are only used to facilitate the description of the embodiments and simplify the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore should not be understood as a limitation of this specification.

[0036] The data involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection of relevant data complies with the relevant laws, regulations and standards of relevant countries and regions.

[0037] This specification first provides a distributed data mining task scheduling method for implementing the scheduling of distributed data mining tasks.

[0038] Before introducing the technical solution of this embodiment, the application scenario of this embodiment is introduced.

[0039] Distributed data mining uses the advantages of distributed computing to process large-scale data sets, especially for those situations where the data volume is huge and difficult to process on a single machine. It can process large-scale data sets that exceed the processing capacity of a single computer. Parallel processing is used to accelerate the data mining process and improve the processing speed. It has strong fault tolerance. When a node fails, other nodes can continue to work to ensure the continuity of the data mining process. It has high scalability. As the data grows, the processing capacity can be expanded by adding more nodes. At the same time, through distributed storage, low-cost commodity-grade hardware can be used to store and process data. The main implementation process of distributed data mining can be divided into the following stages: data preparation stage, distributed data storage, distributed data mining algorithm stage, result aggregation stage, model evaluation and application stage, monitoring and maintenance stage.

[0040] The data collection phase obtains raw data from various source systems, which may include databases, log files, sensor data, etc. Merge data from different sources into a unified data set. Remove incomplete, erroneous or inconsistent data records. Convert data into a form suitable for analysis, such as standardized numerical values, encoded categorical variables, etc. The distributed data storage phase stores the data set in multiple computing nodes in a distributed manner, which can be partitioned according to the key value range, hash value, etc. of the data. In order to improve availability and fault tolerance, some key data may be replicated to multiple nodes. Maintain metadata management information about data distribution and storage location for subsequent processing. The distributed data mining algorithm phase selects appropriate mining algorithms according to the problem type, such as classification, clustering, association rule learning, etc. For distributed environments, traditional algorithms may need to be modified to adapt to the characteristics of data distribution. Split the algorithm into tasks that can be executed in parallel and assign them to different nodes for processing. Intermediate calculation results may need to be exchanged between nodes to complete the global task.

[0041] In the result aggregation phase, after each node completes its own data mining task, it will get some local results. Aggregate the local results of all nodes to form the final global result. Use the validation set to verify the mined model to ensure its accuracy and generalization ability. In the model evaluation and application phase, the test set is used to evaluate the final model, including indicators such as accuracy, recall rate, and F1 score. Adjust and optimize the model according to the evaluation results. Deploy the optimized model to the production environment for real-time or batch data processing. In the monitoring and maintenance phase, continuously monitor the performance indicators of the system, such as processing speed, resource utilization, etc. When the system fails, it can automatically or manually restore data and services. As data changes and technology develops, the system is regularly updated to maintain its effectiveness. Through the above steps, distributed data mining can run efficiently on large-scale data sets while processing real-time streaming data, making data analysis more flexible and powerful.

[0042] Since this manual involves some professional terms, these professional terms will be introduced below.

[0043] Probability of probability distribution

[0044] It represents an approximate estimate of the distribution probability, that is, a probability distribution that contains errors or uncertainties to a certain extent. Because the cost of the distribution probability of the values ​​of the statistical column is high, a large number of database query operations are required, especially for database systems with more columns, and its execution efficiency is very low. And the technical solution provided in this specification does not require the accuracy of the distribution probability. Therefore, an approximate estimate of the distribution probability can be used. The probable distribution probability can be obtained by rough observation of the data or based on a limited sample, without the need for an accurate distribution based on a large amount of detailed data. The probable distribution probability contains a certain degree of subjective judgment or prediction based on the model, so it has a certain degree of uncertainty. When the data quality is not high or there are errors in the data collection process, the obtained probable distribution probability will have a certain degree of ambiguity. However, the errors and ambiguity of the probability distribution basically do not affect the implementation of the technical solution provided in this specification.

[0045] Distribution probability function

[0046] The distribution probability function is one of the basic concepts of probability theory. In practical problems, we often need to study the probability that a random variable ξ takes a value less than a certain value x. This probability is a function of x. This function is called the distribution probability function of the random variable ξ, or distribution function for short. It can determine the probability of a random variable falling into any range.

[0047] The distributed data mining task scheduling method provided in this specification can be applied to application scenarios such as e-commerce recommendation systems, social media analysis, medical and health data analysis, financial risk management, and Internet of Things (IoT) data analysis.

[0048] Taking the e-commerce recommendation system as an example, suppose a large e-commerce platform has millions or even hundreds of millions of user activity records every day. In order to provide users with a better shopping experience, the platform needs to analyze users' purchase history, browsing habits, search keywords, average consumption amount, age, education, gender and other factors. Traditional single-machine data mining methods are inefficient in this case, so it is necessary to adopt distributed data mining technology. Distributed data mining technology specifically includes: data collection, collecting data from various sources, such as databases, log files, etc. Data preprocessing, cleaning data, removing duplicates, missing value processing, etc. Parallel mining algorithm, executing data mining algorithms in parallel on each node, such as collaborative filtering, clustering analysis, etc. Parallel execution of data mining algorithms is to execute data processing tasks 12, assign multiple data processing tasks 12 to multiple distributed nodes 13 for execution, so that distributed nodes 13 can basically complete data processing tasks at the same time. The process is recorded as a scheduling process. Result integration, summarize the results of each node to obtain the final recommendation list. In this way, distributed data mining technology can significantly improve the performance and accuracy of the recommendation system, thereby improving user experience. Example 1

[0049] This embodiment provides a distributed data mining task scheduling method. Figure 1 , which is a schematic diagram of an application scenario of this embodiment. In this embodiment, after the target data mining task 11 is divided into a plurality of data processing tasks 12 that can be executed in parallel, the data processing tasks 12 are scheduled, so as to allocate the data processing tasks 12 to the distributed nodes 13 for execution.

[0050] Specifically, still taking the e-commerce recommendation system as an example, the task of distributed data mining is to find out the influence of the consumer's historical average monthly consumption amount, age, and education level on the preference for the product of fountain pens. The distributed data mining task needs to dig out such a rule, according to which the index of the recommended fountain pen can be obtained based on the consumer's average monthly consumption amount, age, and education level. That is, assuming that the consumer has entered the search keyword fountain pen, the task of the e-commerce recommendation system is to obtain the index of the recommended fountain pen based on the consumer's average monthly consumption amount, age, and education level, and recommend it to the consumer.

[0051] Please refer to the attached Figure 2 , a distributed data mining task scheduling method, comprising the steps of:

[0052] Step 101) Read multiple data processing tasks 12 that can be executed in parallel divided from the target data mining task 11.

[0053] Obtain the order data containing pens within half a year, as well as the age and education data of consumers. Perform data preprocessing, specifically including the following preprocessing: add age and education to the corresponding order data, so that the order data directly contains the age and education of consumers. Store the order data in a database system of several nodes. For example, store it on 3 nodes. The order data can be divided according to time, such as the first node stores the order data from January to February, the second node stores the order data from March to April, and the third node stores the order data from May to June. These three nodes constitute a distributed database, that is, a logical database. Assume that the database name is db-order, and assume that the order data in the database db-order is stored in the table t-order. The target data mining task 11 is to obtain the correlation between the average monthly consumption amount, age, education, and the indicators of the preferred pens of consumers based on the order data containing pens within half a year (here, the order data after adding age and education). The recommended indicators of pens include the brand, price range, and average monthly sales range of pens. The target data mining task 11 can be further described as obtaining the consumer's average monthly consumption amount, age, and education level based on the order data containing pens within half a year (here, the order data after age and education level are added), and inferring the brand, price range, and average monthly sales range of the pen that the consumer prefers.

[0054] The method for dividing the read target data mining task 11 into multiple data processing tasks that can be executed in parallel is: by dividing the consumer set according to the average monthly consumption amount, age, education level, order creation date, and order number, the preferred pen brand, preferred pen price range, and preferred average monthly pen sales range of each consumer set are obtained. Exemplarily, the three data processing tasks obtained by division are as follows: a: based on the five preferred pen brands of consumers who were created from January 1 to January 30, with the last digit of the order number being 0-4, the average monthly consumption amount being 5,000-10,000 yuan, the education level being bachelor degree or above, and the age being 25 and below; b: based on the five preferred pen brands of consumers who were created from January 1 to January 30, with the last digit of the order number being 0-4, the average monthly consumption amount being 5,000-10,000 yuan, the education level being bachelor degree or above, and the age being 25 to 40; c: based on the five preferred pen brands of consumers who were created from January 1 to January 30, with the last digit of the order number being 0-4, the average monthly consumption amount being 5,000-10,000 yuan, the education level being bachelor degree or above, and the age being 40 and above. According to the above three data processing tasks 12, it is possible to obtain five preferred brands of pens among consumers of different age groups with an average monthly consumption amount of 5,000 yuan to 10,000 yuan and an education level of bachelor degree or above. Of course, the divided data processing tasks 12 also include the ability to mine other preferences, for example, mining the price range of pens preferred by consumers of different age groups, and the average monthly sales range of pens preferred by consumers of different education levels (i.e., whether they belong to mass products or niche products according to the audience). Of course, it also includes the rules mined from different order data, such as the corresponding rules mined from the order data created from January 1 to January 30, with the last digit of the order number being 5-9.

[0055] After these divided data processing tasks 12 are completed, they are finally merged to obtain the result of the target data mining task 11. For example, the data processing task a is based on the five pen brands preferred by consumers with an order number ending in 0-4, an average monthly consumption amount of 5,000 yuan to 10,000 yuan, a bachelor's degree or above, and an age of 25 years and below, which are created from January 1 to January 30, and the data processing task d is based on the five pen brands preferred by consumers with an order number ending in 5-9 ... and the data processing task d is based on the five pen brands preferred by consumers with an order number ending in 5-9, an average monthly consumption amount of 5,000 yuan to 10,000 yuan, a bachelor's degree or above, and an age of 25 years and below, and the data processing task d Then, based on the number of orders corresponding to the 10 pen brands, 5 are selected so that there are still 5 pen brands after the fusion.

[0056] Similarly, by merging the order data created from January 1 to January 30 with the order data created from February 1 to February 30, and then merging it with the order data created in March, April, May, and June, we can obtain the five pen brands preferred by consumers who are 25 years old or younger, have an average monthly consumption of 5,000-10,000 yuan, have a bachelor's degree or above, and are based on order data within six months.

[0057] Then, it is integrated with other intervals of average monthly consumption, education level, and age group. Finally, the result of target data mining task 11 is obtained. The recommended intervals for average monthly consumption are 0-5000 yuan, 5000 yuan-10000 yuan, and greater than 10000 yuan; the education level is divided into high school and below, junior college, and undergraduate and above.

[0058] Step 102) Read the data processing task code, identify the database query statement in the code, and execute the database query statement to obtain result data.

[0059] For example, the snippet of the database query statement in the code of data processing task a is as follows:

[0060] import mysql.connector

[0061] from mysql.connector import Error

[0062] try:

[0063] # Connect to MySQL database

[0064] connection = mysql.connector.connect(

[0065] host='your_host', # database server address

[0066] database='db-order', # database name

[0067] user='your_username', # username

[0068] password='your_password' # Password )

[0070] if connection.is_connected():

[0071] cursor = connection.cursor(dictionary=True)

[0072] SELECT

[0073] t_order.order_id,

[0074] t_order.customer_id,

[0075] t_order.product_brand_code, / / This field stores the brand code of the pen

[0076] t_order.creation_date,

[0077] t_order.amount_spent,

[0078] t_customer.education_level,

[0079] t_customer.age

[0080] FROM

[0081] db_order.t_order AS t_order

[0082] JOIN

[0083] db_order.t_customer AS t_customer ON t_order.customer_id = t_customer.customer_id

[0084] WHERE

[0085] t_order.creation_date BETWEEN '2024-01-01' AND '2024-01-30' / / created between January 1 and January 30

[0086] AND t_order.order_id LIKE '%[0-4]' / / The last digit of the order number is 0-4

[0087] AND t_order.amount_spent BETWEEN 5000 AND 10000 / / Average monthly spending is between 5000 yuan and 10000 yuan

[0088] AND t_customer.education_level IN ('Bachelor', 'Master', 'PhD') / / Bachelor degree or above

[0089] AND t_customer.age<= 25 / / Age 25 and below

[0090] AND t_order.product_name = 'Pen'; / / The product of the order is a pen

[0091] except Error as e:

[0092] print("Error while connecting to MySQL", e)

[0093] finally:

[0094] if (connection.is_connected()):

[0095] cursor.close()

[0096] connection.close()

[0097] print("MySQL connection is closed")

[0098] Correspondingly, the database query statements in data processing task b are basically the same, with only "t_order.creation_date BETWEEN '2024-02-01' AND '2024-02-29'" being different. The same is true for data processing tasks c and d.

[0099] Assume that the database query statement for executing data processing task a obtains the result data as follows:

[0100] [{'orderid':'1234','customerid':'C1001','productbrandcode':'PB001','creationdate':'2024-01-05','amountspent':6000,'educationlevel':'Bachelor','age':24}, {'orderid':'1235','customerid':'C1002','productbrandcode':'PB002','creationdate':'2024-01-10','amountspent':5500,'educationlevel':'Master','age':23},…].

[0101] Step 103) Convert the database query statement into a result data reference code.

[0102] First, assign the result data obtained in step 102) to a newly declared array variable, or assign the whole data as a string to a string variable. Note that the length cannot exceed the maximum allowed length of the variable. Specifically, the code for converting to result data reference is:

[0103] [Previous sentence…]

[0104] result=[{'orderid':'1234','customerid':'C1001','productbrandcode':'PB001','creationdate':'2024-01-05','amountspent':6000,'educationlevel':'Bachelor','age': 24},{'orderid':'1235','customerid':'C1002','productbrandcode':'PB002','creationdate':'2024-01-10','amountspent':5500,'educationlevel':'Master','age':23},…]

[0105] [Subsequent statement…]

[0106] In the subsequent statements, the result array is operated to implement the operation on the result data. Exemplarily, the subsequent statements include the following parts:

[0107] brand_order_counts = {}

[0108] for result in results:

[0109] brand = result['product_brand_code']

[0110] if brand in brand_order_counts:

[0111] brand_order_counts[brand] += 1

[0112] else:

[0113] brand_order_counts[brand] = 1

[0114] sorted_brands = sorted(brand_order_counts.items(), key=lambda item:item[1], reverse=True)

[0115] top_brands = sorted_brands[:5];

[0116] Finally, the five preferred brands of pens by consumers who were created from January 1 to January 30, whose order numbers ended with 0-4, whose monthly average spending amount was RMB 5,000-10,000, whose education level was bachelor's degree or above, and whose age was 25 or below are stored in the array variable top_brands. It is worth noting that the subsequent statements here are only for example.

[0117] Step 104) Obtaining an estimated time consumption of the data processing task according to the result data.

[0118] Assume that the length of the result array of the data processing task a obtained in step 103) is 12674, that is, there are 12674 result data. The estimated time required for the data processing task a can be obtained based on the product of the estimated time required to execute each data and 12674, as well as the estimated time of other statements.

[0119] Step 105) Based on the estimated time consumption, the plurality of data processing tasks 12 are distributed to a plurality of nodes for execution, so that the time difference between the plurality of distributed nodes 13 in completing the distributed data processing tasks is minimized.

[0120] For example, the length of the result array of data processing task a is 12674, the length of the result array of data processing task b is 31821, and the length of the result array of data processing task c is 23284. When these three data processing tasks 12 are assigned to two distributed nodes 13 for execution, data processing tasks a and data processing tasks c are assigned to one distributed node 13 for execution, and data processing task b is assigned to one distributed node 13 for execution, so that the time difference between the two distributed nodes 13 in completing the assigned data processing tasks can be minimized. Therefore, when all data processing tasks 12 divided from the entire target data mining task 11 are merged for results, the waiting time for some unfinished data processing tasks is minimized.

[0121] The technical problem addressed by this embodiment is that when the execution time of a data processing task is related to the amount of data that meets the conditions in the database, when allocating data processing tasks 12, it is impossible to obtain the estimated time more accurately by only reading the code, so it is difficult to perform balanced data processing task scheduling from the perspective of time consumption. For example, from the perspective of code alone, the estimated time consumption of data processing tasks a, data processing tasks b, and data processing tasks c will be basically the same. However, the actual time consumption of data processing tasks a, data processing tasks b, and data processing tasks c is mainly determined by the corresponding number of result rows in the database.

[0122] When the technical solution described in this embodiment is adopted, the database query statement can be pre-executed, that is, before the data processing task 12 is allocated and scheduled, the distributed node 13 responsible for scheduling (or one or more designated distributed nodes 13, or one or more dynamically allocated distributed nodes 13) first extracts the database query statement and executes it. The result data is directly embedded in the code, such as the aforementioned array variable result assignment statement. Of course, when the number of rows of result data is large, it can also be output separately to a text file and introduced into the code. Those skilled in the art can choose to implement it. As an example, the method for converting the database query statement into the result data reference code includes the steps of:

[0123] Counting the number of data processing tasks corresponding to each result data;

[0124] When the number exceeds a first preset threshold, a result data file is generated, a database query statement in the data processing task code is replaced with a reference statement to the result data file, and a data screening statement for the result data file is generated according to a query condition of the database query statement;

[0125] When the number does not exceed the first preset threshold, a result data record statement is generated to replace the database query statement, and a data screening statement for the data record statement is generated according to the query condition of the database query statement.

[0126] When the number of data processing tasks corresponding to the result data exceeds the first preset threshold value, it means that the data processing tasks that need to use the result data are more, and a result data file is generated at this time, such as a txt text file, a word text file or a binary file, and a reference to the result data file is added in the data processing task that needs to use the result data. It is possible to save the storage space occupied by the result data and the network occupied space during transmission. When the data processing tasks that need to use the result data are less, the result data record statement is directly generated and embedded in the code of the data processing task, such as the aforementioned array variable result assignment statement. Although the volume of the code will be increased, the database connection and query times that can be reduced will bring about the improvement of data processing efficiency.

[0127] The new technical effects that can be achieved by this embodiment include: being able to more accurately obtain the estimated time consumption of data processing tasks, thereby being able to better schedule distributed data mining tasks and improve the execution efficiency of distributed data mining tasks; being able to execute database query statements more centrally, thereby being able to execute multiple data processing task database query statements in one database connection, that is, being able to reduce the number of times the database needs to establish connections and alleviate database pressure.

[0128] Example 2

[0129] In this embodiment, the method of reading the data processing task code, identifying the database query statement in the code, and executing the database query statement to obtain result data includes the following steps:

[0130] Reading a plurality of the data processing task codes at one time, and extracting all database query statements in the codes;

[0131] Identify database query statements for the same data table and group them together, merge the query conditions of the database query statements in the same group, and thus merge the database query statements in the same group, which is called fused query statements;

[0132] Connect to the database, execute the fusion query statement, and obtain fusion query result data. The rest is the same as in Example 1.

[0133] Exemplarily, the database query statements of data processing task a, data processing task b, and data processing task c are read at one time. Since the database query statements of data processing task a, data processing task b, and data processing task c are queries on the same data table, these database query statements can be merged.

[0134] The merged database query statement is:

[0135] SELECT

[0136] t_order.order_id,

[0137] t_order.customer_id,

[0138] t_order.product_brand_code, / / This field stores the brand code of the pen

[0139] t_order.creation_date,

[0140] t_order.amount_spent,

[0141] t_customer.education_level,

[0142] t_customer.age

[0143] FROM

[0144] db_order.t_order AS t_order

[0145] JOIN

[0146] db_order.t_customer AS t_customer ON t_order.customer_id = t_customer.customer_id

[0147] WHERE

[0148] t_order.creation_date BETWEEN '2024-01-01' AND '2024-01-30' / / created between January 1 and January 30

[0149] AND t_order.order_id LIKE '%[0-4]' / / The last digit of the order number is 0-4

[0150] AND t_order.amount_spent BETWEEN 5000 AND 10000 / / Average monthly spending is between 5000 yuan and 10000 yuan

[0151] AND t_customer.education_level IN ('Bachelor', 'Master', 'PhD') / / Bachelor degree or above

[0152] AND (t_customer.age<= 25 OR (t_customer.age>25 AND t_customer.age<= 40) OR t_customer.age>40) / / Multiple age groups

[0153] AND t_order.product_name = 'Pen'; / / The product of the order is a pen

[0154] Among them, "AND ( t_customer.age<= 25 OR ( t_customer.age>25 AND t_customer.age<= 40) OR t_customer.age>40)" realizes the fusion query result data of the database query statements of data processing task a, data processing task b, and data processing task c at one time. Similarly, the fusion query result data is assigned to an array variable.

[0155] Then, the number of data rows corresponding to each data processing task 12 is counted according to this numerical variable, so as to obtain the estimated time consumption of the data processing task.

[0156] Please refer to the attached Figure 3 In this embodiment, the method for fusing query conditions of database query statements in the same group includes the steps of:

[0157] Step 201) When the conditions of multiple columns in the query conditions are different, multiple database query statements are executed in one database connection. When the condition of only one column in the query conditions is different, the numerical range condition and the numerical value condition of the same column in the query conditions are combined to obtain a combined numerical condition by fusing the numerical range condition and the numerical value condition.

[0158] Exemplarily, when only the condition of t_customer.age is different in the database query statement, the fusion of the numerical range condition is to fuse t_customer.age<= 25, t_customer.age>25 AND t_customer.age<= 40, and t_customer.age>40 into t_customer.age<= 25 OR ( t_customer.age>25 AND t_customer.age<= 40) OR t_customer.age>40. Then, after the result data reference code, add the corresponding filter statement. Exemplarily, after the fusion query statement is executed, the fusion query result data is assigned to the array variable result, and then in the loop body of the processing of the result row (i.e., the for loop body), add the statement if t_customer.age>25 then skip this loop to implement the addition of the filter statement. Or, the addition of the filter statement is implemented according to other technologies already disclosed in the art.

[0159] For example, when only the condition of t_order.product_brand_code is different in the database query statement, the fusion of the numerical value conditions is to fuse t_order.product_brand_code='012' and t_order.product_brand_code='424' into t_order.product_brand_code='012' OR t_order.product_brand_code='424'.

[0160] Step 202) Perform a union operation on the label value conditions for the same column in the query conditions to obtain a fused label condition.

[0161] For example, when only the condition of t_customer.education_level is different in the database query statement, the label value conditions t_customer.education_level IN ('HighSchool', 'MiddleSchool', 'ElementarySchool') and t_customer.education_level IN ('Bachelor', 'Master', 'PhD') are merged to obtain the fused label condition t_customer.education_level IN ('HighSchool', 'MiddleSchool', 'ElementarySchool','Bachelor', 'Master', 'PhD').

[0162] Step 203) Obtain the possible value range of the column corresponding to the fused numerical condition, and when the ratio of the numerical range corresponding to the fused numerical condition to the possible value range exceeds a preset ratio threshold, delete the fused numerical condition.

[0163] For example, when the value range of t_customer.age in the fused data condition has covered <= 25, >25 AND <= 40, >40, the value range corresponding to the actual fused data condition accounts for 100% of the possible value range, and t_customer.age can no longer be filtered. That is, the condition setting for the column t_customer.age in the database query statement is directly deleted.

[0164] At this time, the corresponding part of the fused database query statement is:

[0165] WHERE

[0166] t_order.creation_date BETWEEN '2024-01-01' AND '2024-01-30' / / created between January 1 and January 30

[0167] AND t_order.order_id LIKE '%[0-4]' / / The last digit of the order number is 0-4

[0168] AND t_order.amount_spent BETWEEN 5000 AND 10000 / / Average monthly spending is between 5000 yuan and 10000 yuan

[0169] AND t_customer.education_level IN ('Bachelor', 'Master', 'PhD') / / Bachelor degree or above

[0170] AND t_order.product_name = 'Pen'; / / The product of the order is a pen

[0171] Accordingly, the possible value label set of the column corresponding to the fusion label condition is obtained, and when the fusion label condition covers the possible value label set, the fusion label condition is deleted. Exemplarily, if the multiple data processing tasks 12 read include t_customer.education_level IN ('HighSchool', 'MiddleSchool', 'ElementarySchool')) and t_customer.education_level = 'AssociateDegree', and the screening conditions of other columns are the same, since all the academic qualifications labels are covered, the academic qualifications can no longer be screened, that is, the fusion label condition is deleted.

[0172] On the other hand, in another embodiment, before deleting the fusion numerical condition, a preset checklist is read, and the checklist records the probabilities of the probable distribution of the columns;

[0173] Obtaining the probable distribution probability of the column corresponding to the fusion numerical condition according to the lookup table;

[0174] When the probability of the probable distribution corresponding to the numerical range corresponding to the fused data condition is lower than a preset threshold, the fused numerical condition is prevented from being deleted; when the probability of the probable distribution corresponding to the numerical range corresponding to the fused data condition is not lower than the preset threshold, the fused numerical condition is deleted.

[0175] Exemplarily, the check table records that the distribution of the column t_customer.age is uniformly distributed, and the probable distribution probability of uniform distribution is easy to obtain. Before this distributed data mining task, the average monthly consumption amount distribution probability of consumers was obtained based on the consumer information registered in the e-commerce, so that the probable distribution probability of the column t_order.amount_spent can be recorded in the check table, assuming that it is {'<=5000':45%,'>5000 AND<=10000':40%,'>10000':15%}. Assuming that the preset threshold is 80%, when the fused data condition is t_order.amount_spent>5000, since the probable distribution probability corresponding to the numerical range corresponding to the fused data condition is 55%, which is lower than the preset threshold of 80%, the operation of deleting the fused numerical condition is not performed at this time.

[0176] The method of presetting the checklist is recommended to calculate and obtain the data generated by e-commerce before the distributed data mining task is carried out, and use it as public data. Of course, on the other hand, other methods can also be used to obtain it. For example, the method of presetting the checklist includes the steps of:

[0177] Preset several distribution probability functions;

[0178] Selecting a distribution probability function from a plurality of the distribution probability functions as the distribution probability function of the column according to the attribute of the column;

[0179] The probable distribution probability of the column is obtained according to the distribution probability function.

[0180] Reading the data generated by e-commerce and performing calculations requires computing power. You can manually or through a large model, directly based on experience or knowledge learned from the large model, select a distribution probability function from several distribution probability functions as the distribution probability function of the column. Just make a rough estimate. For example, the average monthly consumption amount t_order.amount_spent column can be directly set to conform to the Zhengtai distribution, and further calculate each amount interval to obtain the probable distribution probability.

[0181] On the other hand, in another embodiment, please refer to the attached Figure 4 The method for obtaining the estimated time consumption of the data processing task according to the result data comprises the steps of:

[0182] Step 301) Read a preset statement estimated time consumption table, and obtain the estimated time consumption of each statement of the data processing task code according to the statement estimated time consumption table.

[0183] Step 302) Obtain the execution times of each statement of the data processing task code according to the number of rows of the fusion query result data.

[0184] Step 303) According to the estimated time consumption and execution times of each statement, the result data is obtained to obtain the estimated time consumption of the data processing task.

[0185] Exemplarily, the code of data processing task a includes import statements, connection statements, SELECT statements, for loop statements, sorting statements sorted_brands, and assignment statements of top_brands. The estimated time consumption of import statements, sorting statements sorted_brands, and assignment statements of top_brands are all recorded as 0.1ms in the statement estimated time consumption table. Since the connection statement and SELECT statement require the response of the database, their estimated time consumption is recorded as 0.3ms. The for loop body statement is the product of the number of statements in the loop body and 0.1ms, and then multiplied by the number of loops. The number of for loop body statements in the code of data processing task a is 2, so the estimated time consumption of for loop body statements in the code of data processing task a is 0.1ms×2×12674, so that the estimated time consumption of data processing task a can be obtained.

[0186] Compared with Example 1, the new technical effects that can be achieved by this embodiment include: further reducing database connections and database query times by fusing query statements, alleviating database load pressure, reducing the time waiting for database response in data processing tasks, and improving the execution efficiency of data processing tasks.

[0187] On the other hand, this specification provides a distributed data mining task scheduling system, see the attached Figure 5 ,include:

[0188] A reading module 21 reads a plurality of data processing tasks 12 that can be executed in parallel divided from a target data mining task 11;

[0189] The pre-execution module 22 reads the data processing task code, identifies the database query statement in the code, and executes the database query statement to obtain result data;

[0190] A conversion module 23 converts the database query statement into a result data reference code;

[0191] An estimation module 24 is used to obtain an estimated time consumption of the data processing task according to the result data;

[0192] The scheduling module 25 distributes the plurality of data processing tasks 12 to a plurality of nodes for execution according to the estimated time consumption, so as to minimize the time difference for the plurality of distributed nodes 13 to complete the distributed data processing tasks.

[0193] See also Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of this specification is shown.

[0194] like Figure 6 As shown, the electronic device 1100 may include: at least one processor 1101, at least one network interface 1104, a user interface 1103, a memory 1105 and at least one communication bus 1102. Among them, the communication bus 1102 can be used to realize the connection and communication of the above-mentioned components. Among them, the user interface 1103 may include a button, and the optional user interface may also include a standard wired interface and a wireless interface. Among them, the network interface 1104 may include but is not limited to a Bluetooth module, an NFC module, a Wi-Fi module, etc. Among them, the processor 1101 may include one or more processing cores. The processor 1101 uses various interfaces and lines to connect the various parts of the entire electronic device 1100, and executes various functions and processes data of the routing device 1100 by running or executing instructions, programs, code sets or instruction sets stored in the memory 1105, and calling data stored in the memory 1105. Optionally, the processor 1101 can be implemented in at least one hardware form of DSP, FPGA, and PLA. The processor 1101 can integrate one or a combination of CPU, GPU, modem, etc. Among them, the CPU mainly handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content that needs to be displayed on the display; and the modem is used to handle wireless communications.

[0195] It is understandable that the above-mentioned modem may not be integrated into the processor 1101, but may be implemented by a separate chip.

[0196] Among them, the memory 1105 may include RAM or ROM. Optionally, the memory 1105 includes a non-transitory computer-readable medium. The memory 1105 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 1105 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 1105 may also be at least one storage device located away from the aforementioned processor 1101. The memory 1105 as a computer storage medium may include an operating system, a network communication module, a user interface module and an application. The processor 1101 may be used to call the application stored in the memory 1105 and execute the methods in the above-mentioned multiple embodiments.

[0197] The embodiments of this specification also provide a computer-readable storage medium, which stores instructions, and when the instructions are executed on a computer or a processor, the computer or the processor executes the multiple steps in the above embodiments. If the components of the above electronic device are implemented in the form of software functional units and sold or used as independent products, they can be stored in the computer-readable storage medium.

[0198] The embodiments of this specification also provide a computer program product, including a computer program, which implements multiple steps in the above embodiments when executed by a processor.

[0199] In the absence of conflict, the technical features in this embodiment and implementation scheme can be combined arbitrarily.

[0200] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes a plurality of computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of this specification is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes multiple available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital versatile disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)), etc.

[0201] When implemented by hardware or firmware, the aforementioned method flow is programmed into the hardware circuit to obtain the corresponding hardware circuit structure and realize the corresponding function. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit, and its logic function is determined by the user programming the device. The designer programs by himself to "integrate" a digital system on a PLD, without asking a chip manufacturer to design and make a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when writing program development, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL), and HDL is not just one, but many. Those skilled in the art should also be aware that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit to easily obtain the hardware circuit that implements the logic method flow.

[0202] The embodiments described above are merely preferred embodiments of this specification and are not intended to limit the scope of this specification. Without departing from the design spirit of this specification, various modifications and improvements made to the technical solutions of this specification by ordinary technicians in this field should fall within the scope of protection determined by the claims of this specification.

Claims

1. A distributed data mining task scheduling method, characterized in that: Includes steps: Read multiple data processing tasks that can be executed in parallel divided by the target data mining task; Reading the code of the data processing task, identifying the database query statement in the code, and executing the database query statement to obtain result data; Converting the database query statement into a result data reference code; Obtaining an estimated time consumption of the data processing task according to the result data; Allocate the plurality of data processing tasks to a plurality of nodes for execution according to the estimated time consumption, so as to minimize the time difference of the plurality of distributed nodes in completing the allocated data processing tasks; The method of reading the code of the data processing task, identifying the database query statement in the code, and executing the database query statement to obtain result data comprises the steps of: Reading the codes of the plurality of data processing tasks at one time, and extracting all database query statements in the codes; Identify database query statements for the same data table and group them together, merge the query conditions of the database query statements in the same group, and thus merge the database query statements in the same group, which is called fused query statements; Connecting to the database, executing the fusion query statement, and obtaining fusion query result data; The method for obtaining the estimated time consumption of the data processing task according to the result data comprises the steps of: Reading a preset statement estimated time consumption table, and obtaining an estimated time consumption of each statement of the code of the data processing task according to the statement estimated time consumption table; Obtaining the execution times of each statement of the code of the data processing task according to the number of rows of the fusion query result data; The result data is obtained based on the estimated time consumption and execution times of each statement to obtain the estimated time consumption of the data processing task.

2. A distributed data mining task scheduling method according to claim 1, characterized in that: The method for converting the database query statement into a result data reference code comprises the steps of: Counting the number of the data processing tasks corresponding to each result data; When the number exceeds a first preset threshold, a result data file is generated, a database query statement in the code of the data processing task is replaced with a reference statement to the result data file, and a data screening statement for the result data file is generated according to a query condition of the database query statement; When the number does not exceed the first preset threshold, a result data record statement is generated to replace the database query statement, and a data screening statement for the data record statement is generated according to the query condition of the database query statement.

3. A distributed data mining task scheduling method according to claim 1 or 2, characterized in that: The method for fusing query conditions of database query statements in the same group comprises the steps of: When the conditions of multiple columns in the query conditions are different, multiple database query statements are executed in one database connection. When the conditions of only one column in the query conditions are different, the value range condition and the value condition of the same column in the query conditions are combined to obtain a combined value condition by combining the value range condition and the value condition. Perform a union operation on the label value conditions of the same column in the query conditions to obtain the fused label condition; The possible value range of the column corresponding to the fused numerical condition is obtained, and when the ratio of the numerical range corresponding to the fused numerical condition to the possible value range exceeds a preset ratio threshold, the fused numerical condition is deleted.

4. A distributed data mining task scheduling method according to claim 3, characterized in that: Before deleting the fusion numerical condition, read a preset checklist, the checklist recording the probabilities of the probable distribution of the columns; Obtaining the probable distribution probability of the column corresponding to the fusion numerical condition according to the lookup table; When the probability distribution corresponding to the numerical range corresponding to the fused numerical condition is lower than a preset threshold, the fused numerical condition is prevented from being deleted; when the probability distribution corresponding to the numerical range corresponding to the fused numerical condition is not lower than the preset threshold, the fused numerical condition is deleted; The method for presetting the checklist includes the following steps: Preset several distribution probability functions; Selecting a distribution probability function from a plurality of the distribution probability functions as the distribution probability function of the column according to the attribute of the column; The probable distribution probability of the column is obtained according to the distribution probability function.

5. A distributed data mining task scheduling system, characterized in that: include: A reading module reads multiple data processing tasks that can be executed in parallel divided by the target data mining task; A pre-execution module reads the code of the data processing task, identifies the database query statement in the code, and executes the database query statement to obtain result data; A conversion module converts the database query statement into a result data reference code; An estimation module, for obtaining an estimated time consumption of the data processing task according to the result data; A scheduling module, which distributes the plurality of data processing tasks to a plurality of nodes for execution according to the estimated time consumption, so as to minimize the time difference of the plurality of distributed nodes in completing the distributed data processing tasks; The method of reading the code of the data processing task, identifying the database query statement in the code, and executing the database query statement to obtain result data comprises the steps of: Reading the codes of the plurality of data processing tasks at one time, and extracting all database query statements in the codes; Identify database query statements for the same data table and group them together, merge the query conditions of the database query statements in the same group, and thus merge the database query statements in the same group, which is called fused query statements; Connecting to the database, executing the fusion query statement, and obtaining fusion query result data; The method for obtaining the estimated time consumption of the data processing task according to the result data comprises the steps of: Reading a preset statement estimated time consumption table, and obtaining an estimated time consumption of each statement of the code of the data processing task according to the statement estimated time consumption table; Obtaining the execution times of each statement of the code of the data processing task according to the number of rows of the fusion query result data; The result data is obtained based on the estimated time consumption and execution times of each statement to obtain the estimated time consumption of the data processing task.

6. An electronic device, characterized in that including a processor and a memory; The processor is connected to the memory; The memory is used to store executable program code; The processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to execute the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

8. Computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images