Execution engine determination method, model training method and device

Abstract

This application provides an execution engine determination method, model training method, and apparatus, relating to the field of big data. The method comprises: obtaining an application job for which an execution engine is to be determined; inputting characteristic data of the application job into an execution engine prediction model, the execution engine prediction model being used to calculate, based on the characteristic data of the application job, a probability that the application job is suitable for one of two preset execution engines; and setting a target execution engine for the application job based on the probability, the target execution engine being one of the two preset execution engines. This allows for automatic setting of a suitable execution engine for the application job, thereby improving the execution efficiency of the application job.

Description

Technical Field

[0001] The present application relates to the field of big data, and in particular to a method for determining an execution engine, a model training method, and a device. Background Art

[0002] Hadoop is a software framework that enables distributed processing of large amounts of data. Hadoop can run numerous application jobs. Currently, running a large number of application jobs on Hadoop requires application developers to use their experience to specify an execution engine for each application job. This approach relies heavily on the developer's experience, and sometimes the execution engine specified by the developer for the application job is not appropriate, resulting in low application job execution efficiency. Summary of the Invention

[0003] This application provides an execution engine determination method, a model training method, and an apparatus for automatically setting a suitable execution engine for an application job, thereby improving the execution efficiency of the application job.

[0004] In a first aspect, the present application provides a method for determining an execution engine, the method comprising: obtaining an application job of an execution engine to be determined; inputting characteristic data of the application job into an execution engine prediction model, the execution engine prediction model being used to calculate the probability that the application job is applicable to one of two preset execution engines based on the characteristic data of the application job; and setting a target execution engine for the application job based on the probability, the target execution engine being one of the two preset execution engines.

[0005] The method can be applied to the Hadoop platform, and the two preset execution engines can be Spark (the name of an execution engine) and MapReduce (the name of an execution engine, which can be abbreviated as MR).

[0006] Based on the above scheme, when an application job needs to be executed, the execution engine prediction model can be used to calculate the probability that the application job is suitable for one of the two preset execution engines, and then a suitable target execution engine can be set for the application job based on the calculated probability, thereby automatically setting a suitable execution engine for the application job, which is conducive to improving the execution efficiency of the application job.

[0007] In a second aspect, the present application provides a model training method, which includes: acquiring data, the data including feature data of two preset execution engines, feature data of multiple application jobs, a label of each application job in the multiple application jobs, feature data of resources used by the multiple application jobs, and feature data of resources provided by each tenant in at least one tenant running the multiple applications, the label of each application job is used to indicate the execution engine applicable to each application job; performing model training based on the data to obtain an execution engine prediction model, the execution engine prediction model being used to predict the execution engine applicable to the application job.

[0008] Based on the above scheme, the model is trained by comprehensively considering the characteristics of the application job, the characteristics of the resources used by the application job, the characteristics of the resources provided by each tenant in at least one tenant running the application, and the characteristics of the two preset execution engines, so that a better execution engine prediction model that integrates the above-mentioned characteristic factors can be obtained. The execution engine prediction model can then be used to determine the appropriate execution engine for each application job to be executed, which is conducive to improving the execution efficiency of the application job.

[0009] Optionally, the model training based on the data includes: inputting the characteristic data of the two preset execution engines, the characteristic data of multiple application jobs, the characteristic data of resources used by the multiple application jobs, and the characteristic data of resources provided by each tenant of at least one tenant running the multiple applications into the execution engine prediction model to be trained; processing the characteristic data of the two preset execution engines, the characteristic data of multiple application jobs, the characteristic data of resources used by the multiple application jobs, and the characteristic data of resources provided by each tenant of at least one tenant running the multiple applications through the execution engine prediction model to be trained to obtain a target prediction probability; and training the execution engine prediction model to be trained based on the label of each application job and the target prediction probability.

[0010] Optionally, the execution engine prediction model to be trained processes the characteristic data of the preset two execution engines, the characteristic data of multiple application jobs, the characteristic data of resources used by the multiple application jobs, and the characteristic data of resources provided by each tenant in at least one tenant running the multiple applications to obtain a target prediction probability, including: preprocessing the characteristic data of the preset two execution engines, the characteristic data of multiple application jobs, the characteristic data of resources used by the multiple application jobs, and the characteristic data of resources provided by each tenant in at least one tenant running the multiple applications to obtain preprocessed data; obtaining a prediction value based on the preprocessed data; and obtaining the target prediction probability based on the prediction value.

[0011] Optionally, the preprocessing of the feature data of the two preset execution engines, the feature data of multiple application jobs, the feature data of resources used by the multiple application jobs, and the feature data of resources provided by each tenant of at least one tenant running the multiple applications includes: using zero-score (Z-score) normalization to process data with continuous features to obtain a continuous feature vector; and using one-bit effective one-hot encoding to process data with discrete features to obtain a discrete feature vector.

[0012] Optionally, obtaining the predicted value based on the preprocessed data includes: performing dimensionality reduction processing on the discrete feature vector to obtain a dense vector; calculating the inner product of the dense vector and the continuous feature vector to obtain a first predicted value; and / or inputting the first predicted value into a multilayer perceptron (MLP) to obtain a second predicted value.

[0013] Optionally, obtaining the target predicted probability based on the predicted value includes: inputting the predicted value into a conversion function to obtain the target predicted probability, and the conversion function is used to convert the predicted value into a value between 0 and 1.

[0014] Optionally, the method further includes: periodically acquiring the data; and training the execution engine prediction model based on the data acquired in a most recent period to obtain an updated execution engine prediction model.

[0015] In a third aspect, the present application provides a device for determining an execution engine, the device comprising an acquisition module, an input module and a setting module, the acquisition module being used to acquire an application job of an execution engine to be determined; the input module being used to input characteristic data of the application job into an execution engine prediction model, the execution engine prediction model being used to calculate the probability that the application job is applicable to one of two preset execution engines based on the characteristic data of the application job; the setting module being used to set a target execution engine for the application job based on the probability, the target execution engine being one of the two preset execution engines.

[0016] In a fourth aspect, the present application provides an execution engine determination device, the device comprising a processor, the processor being coupled to a memory and configured to execute a computer program in the memory to implement the method in the first aspect.

[0017] Optionally, the device in the fourth aspect further includes a memory.

[0018] Optionally, the apparatus in the fourth aspect further includes a communication interface, and the processor is coupled to the communication interface.

[0019] In a fifth aspect, the present application provides a model training device, which includes an acquisition module and a training module. The acquisition module is used to acquire data, and the data includes characteristic data of two preset execution engines, characteristic data of multiple application jobs, labels of each application job in the multiple application jobs, characteristic data of resources used by the multiple application jobs, and characteristic data of resources provided by each tenant in at least one tenant running the multiple applications. The label of each application job is used to indicate the execution engine applicable to each application job; the training module is used to perform model training based on the data to obtain an execution engine prediction model, and the execution engine prediction model is used to predict the execution engine applicable to the application job.

[0020] In a sixth aspect, the present application provides a model training device, comprising a processor coupled to a memory and configured to execute a computer program in the memory to implement the method in the second aspect and any possible implementation of the second aspect.

[0021] Optionally, the device in the sixth aspect also includes a memory.

[0022] Optionally, the device in the sixth aspect further includes a communication interface, and the processor is coupled to the communication interface.

[0023] In the seventh aspect, the present application provides a chip system, which includes at least one processor for supporting the implementation of the functions involved in the above-mentioned first and second aspects and any possible implementation method of the second aspect, such as receiving or processing the data involved in the above-mentioned method.

[0024] In one possible design, the chip system further includes a memory, which is used to store program instructions and data, and the memory is located inside or outside the processor.

[0025] The chip system can be composed of chips, or can include chips and other discrete devices.

[0026] In an eighth aspect, the present application provides a computer-readable storage medium, on which a computer program (also referred to as code, or instructions) is stored. When the computer program is executed by a processor, the method in the above-mentioned first aspect and second aspect and any possible implementation of the second aspect is executed.

[0027] In the ninth aspect, the present application provides a computer program product, which includes: a computer program (also referred to as code, or instructions), which, when run, enables the methods in the above-mentioned first and second aspects and any possible implementation of the second aspect to be executed.

[0028] It should be understood that the third to ninth aspects of the present application correspond to the technical solutions of the first and second aspects of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a schematic diagram of a scenario applicable to an embodiment of the present application;

[0030] Figure 2 This is a flow chart provided by an embodiment of the present application;

[0031] Figure 3 This is a flow chart of a model training method provided in an embodiment of the present application;

[0032] Figure 4 is a schematic diagram of a deep neural network provided in an embodiment of the present application;

[0033] Figure 5 This is a flowchart of a method for determining an execution engine provided in an embodiment of the present application;

[0034] Figure 6 is a schematic block diagram of a determination device of an execution engine provided in an embodiment of the present application;

[0035] Figure 7 This is a schematic block diagram of a model training device provided in an embodiment of the present application;

[0036] Figure 8 This is a schematic block diagram of another device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] The technical solution in this application will be described below with reference to the accompanying drawings.

[0038] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0039] To facilitate understanding of the embodiments of the present application, some terms or words involved in the present application are briefly explained below.

[0040] 1. Spark: An execution engine that can be used on the Hadoop platform. Spark is a fast and versatile execution engine designed specifically for large-scale data processing. It can be used to perform a variety of computations, including text processing, machine learning, and statement queries. Statement queries, such as Hibernate Query Language (HQL) and Structured Query Language (SQL), share the advantages of MapReduce, but unlike MapReduce, Spark stores intermediate output results of application jobs in memory, eliminating the need to read or write files to the Hadoop Distributed File System (HDFS).

[0041] 2. MapReduce: A distributed offline execution engine that can be used on the Hadoop platform. MapReduce is well-suited for offline data processing, primarily for large-scale cluster tasks. It can execute application jobs in batches. Intermediate output results of application jobs are stored in HDFS files, requiring frequent reading and writing of HDFS files, resulting in low execution efficiency.

[0042] The embodiments of this application take Spark and MapReduce as examples to introduce the execution engine determination method and model training method.

[0043] 3. Deep neural network (DNN): A deep neural network can be understood as a neural network with multiple hidden layers. Generally speaking, multi-layer neural networks and deep neural networks have the same meaning.

[0044] Based on the position of different layers within a DNN, the neural network can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and all layers in between are hidden layers. Each layer is fully connected, meaning that any neuron in the previous layer is connected to any neuron in the next layer. Although DNNs appear complex, the operation of each layer is actually quite simple. Simply put, it can be expressed as follows: y = δ(wx + b), where x is the input vector, y is the output vector, b is the bias vector, w is the weight vector (also called coefficient), and δ() is the activation function. Each layer simply performs this simple operation on the input vector x to produce the output vector y. Due to the large number of DNN layers, the number of corresponding coefficients w and bias vectors b is considerable.

[0045] 4. Loss function: Usually, because we want the output of a deep neural network to be as close as possible to the desired value, we can compare the current network's predicted value with the desired target value, and then update the weight vector of each layer of the neural network based on the difference between the two (of course, there is usually an initialization process before the first update, which is to pre-configure the parameters for each layer in the deep neural network). For example, if the network's predicted value is too high, the weight vector is adjusted to make it predict a lower value, and this is continuously adjusted until the neural network can predict the desired target value. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value." This is the loss function or objective function, which is an important equation used to measure the difference between the predicted value and the target value. Taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference. Then, the training of the deep neural network becomes a process of minimizing this loss as much as possible.

[0046] The model training method proposed in this application is the training of deep neural networks.

[0047] Hadoop is a software framework that can process large amounts of data in a distributed manner. Figure 1As shown in Figure 1, a large number of application jobs can be run on the Hadoop platform. Currently, each application job that runs a large number of applications on Hadoop requires application developers to specify an execution engine for each application job based on their personal experience, such as Figure 1 As shown, developers need to rely on their personal experience to determine the appropriate execution engine for Application Job 1, Application Job 2, and Application Job 3, among execution engines 1, 2, ..., and L (where L is an integer). For example, if a developer believes based on personal experience that Spark is suitable for executing Application Job 1, then they will set Spark as the execution engine for Application Job 1. If they believe that MapReduce is suitable for executing Application Job 2, then they will set MapReduce as the execution engine for Application Job 2. However, this execution engine setting method relies heavily on the developer's personal experience, and sometimes the execution engine specified by the developer for the application job is not suitable, resulting in low execution efficiency for the application job.

[0048] Therefore, the present application provides a method for determining an execution engine, a model training method and a device. By training an execution engine prediction model, the execution engine prediction model is used to determine the execution engine suitable for each application to be executed, and the determined suitable execution engine is automatically set for each application to be executed.

[0049] It should be noted that the execution engine determination method, model training method and device provided in the embodiments of the present application can be applied to the big data field, and can also be applied to any field other than the big data field, and the present application does not limit this.

[0050] For ease of understanding, in an embodiment of the present application, the process of obtaining an execution engine prediction model by model training is first described, and then the process of determining a suitable execution engine for an application job using the execution engine prediction model is introduced.

[0051] Figure 2 The figure shows the process of model training involved in the embodiment of the present application. Figure 2As shown, sample data can be obtained through data collection. In an embodiment of the present application, the sample data includes the collected feature data, which will be described in detail below and will not be described in detail here. Labels obtained by data annotation. In an embodiment of the present application, labels are used to indicate the execution engine applicable to each application job from which feature data is extracted. The sample data and labels are input into the model for supervised training of the model to obtain an execution engine prediction model, which can then be used to determine a suitable execution engine for the application job to be executed.

[0052] The following describes the model training method with reference to the accompanying drawings.

[0053] Figure 3 It is a flowchart of a model training method provided in an embodiment of the present application.

[0054] like Figure 3 As shown, the model training method 300 includes step 310 and step 320. Each step in the method 300 is described in detail below.

[0055] In step 310 , the computer acquires data.

[0056] Before model training, data collection is required. Taking into account various factors such as the current status of cluster resources, the characteristics of the query statements in the application jobs, and the available resources of the tenant where the application jobs are running, the data that needs to be collected in this embodiment of the application may include feature data of two preset execution engines, feature data of multiple application jobs, feature data of resources used by the multiple application jobs, and feature data of resources provided by each tenant in the at least one tenant running the multiple applications.

[0057] The characteristic data of the two preset execution engines, taking Spark and MapReduce as examples, may include Spark's characteristic data: the total number of query statements executed daily by Spark, the average execution time of each query statement, the maximum execution time, the median execution time of each query statement, etc.; and MapReduce's characteristic data: the total number of query statements executed daily by MapReduce, the average execution time of each query statement, the maximum execution time, the median execution time of each query statement, etc. These characteristic data can be considered as data with continuous characteristics.

[0058] It should be understood that in order to ensure the fairness of the feature comparison between the two execution engines, when collecting the feature data of the two preset execution engines, the two execution engines execute the same batch of query statements. For example, it is possible to select more than 500,000 random query statements from the jobs in the non-low-time job group (which can be understood as a job group with an execution time limit) from 15:00 to 24:00 every day in the standby cluster or the main cluster. The amount of data involved in these query statements must be less than 1 terabyte (TB) (for the safety and stability of the production system, and based on the current accumulated experience, application jobs with more than 1T of data are basically not suitable for using spark). After these query statements are randomly selected every day, MapReduce and Spark will be started at the same time to execute these query statements in parallel. The reason why they must be started at the same time is to ensure the fairness of the feature comparison between the two execution engines. In other words, spark and MapReduce start executing the same batch of query statements at the same time. It should be understood that as long as spark and MapReduce start executing the same batch of query statements at the same time, the specific query statements in the application jobs selected by this application are not limited.

[0059] The characteristic data of an application job may include the total number of records in the input table and the total number of records in the corresponding output table corresponding to each query statement included in the application job (one query statement corresponds to one input table and one output table), the number of records with the largest number of records in the multiple input tables and the number of records with the largest number of records in the multiple output tables corresponding to the multiple query statements included in the application job, the total number of input tables and output tables corresponding to the query statements included in the application job (because one input table or output table is stored in one HDFS file, the total number of input tables and output tables included in the application job may also be the total number of HDFS files corresponding to the query statements included in the application job), the average size, standard deviation, and median of the input tables and output tables corresponding to the query statements included in the application job (which can be understood as the average size, standard deviation, and median of the HDFS files corresponding to the query statements included in the application job), the size of the largest input table or output table corresponding to the query statements included in the application job (which can be understood as the size of the largest HDFS file), the size of the smallest input table or output table corresponding to the query statements included in the application job (which can be understood as the size of the smallest HDFS file), etc. These data can be considered as data with continuous characteristics.

[0060] The characteristic data of an application job may also include the rank of the total number of records in the input table corresponding to the query statements included in an application job relative to the total number of records in the input table corresponding to the query statements included in all application jobs running on the entire Hadoop platform on which the application job is running. For example, if the total number of records in the input table corresponding to the query statements included in the application job is in the top 1%-top 5% of the total number of records in the input table corresponding to the query statements included in all application jobs running on the entire Hadoop platform, the rank is recorded as 1. If the total number of records in the input table corresponding to the query statements included in the application job is in the top 6%-top 10% of the total number of records in the input table corresponding to the query statements included in all application jobs running on the entire Hadoop platform, the rank is recorded as 2, and so on. The characteristic data of an application job may also include the rank of the total number of records in the output table corresponding to the query statements included in the application job relative to the total number of records in the output table corresponding to the query statements included in all application jobs running on the entire Hadoop platform on which the application job is running. For the classification of the rank here, please refer to the above-mentioned classification method. This also includes the rank of the number of input tables corresponding to the query statements included in this application job, and the total number of input and output tables corresponding to the query statements included in all application jobs running on the entire Hadoop platform. Also included is the rank of the number of output tables corresponding to the query statements included in this application job, and the total number of input and output tables corresponding to the query statements included in all application jobs running on the entire Hadoop platform. These data can be considered as data with discrete characteristics.

[0061] Application job feature data may also include the number of stages included in each query statement in an application job, the number of maps and reduce operations required for each query statement, the maximum number of maps and reduce operations involved in each query statement, the average execution time and wait time of each query statement's most recent n executions (n ​​≥ 1, n is an integer, for example, n = 30), the maximum execution time and wait time of each query statement, the minimum execution time and wait time of each query statement, the variance of the query statement execution time in this application job, the variance of the wait time, and the median execution time. This data can be considered as data with continuous features.

[0062] The characteristic data of an application job may also include the rank of the number of stages included in the application job relative to the total number of stages included in all application jobs running on the entire Hadoop platform on which the application job is running, the rank of the number of maps included in the application job relative to the total number of maps included in all application jobs running on the entire Hadoop platform on which the application job is running, and the rank of the number of reduce operations included in the application job relative to the total number of reduce operations included in all application jobs running on the entire Hadoop platform on which the application job is running. For the rank classification herein, refer to the rank classification method described above. This data can be considered as data with discrete characteristics.

[0063] Feature data for an application job can also include the ratio of the execution time of the most recently executed m query statements to the total execution time of the application (m ≥ 1, where m is an integer, e.g., m = 50). For example, if the execution time of one of the most recently executed 50 query statements accounts for 89% of the total execution time of the application job, the execution time feature value of this query statement can be recorded as 89, which can also indicate the importance of this query statement in the application job. Feature data can also include the ratio of the execution time of the most recently executed m query statements to the total execution time of all application jobs running on the Hadoop platform. This data can be considered as data with continuous features.

[0064] The characteristic data of an application job can also include dependency information for each query statement included in the application job. For example, the out-degree of a query statement can be defined as: how many other query statements will be called when the query statement is executed in the application job; the in-degree of a query statement can be defined as: how many query statements in the application job will call the query statement when executed. The out-degree and in-degree of each query statement can be converted into a rank, which represents the sum of the out-degree and in-degree of all query statements in the application job. For the rank classification here, refer to the rank classification method described above. This data can be considered as data with discrete characteristics.

[0065] The characteristic data of the resources used by application jobs can include the total number of query statements currently running on the entire Hadoop platform where the application runs, the resources and memory allocated on the entire Hadoop platform, and the remaining resources and memory. This data can be considered as data with continuous characteristics.

[0066] The characteristic data of resources provided by each tenant in at least one tenant running the multiple applications may include the total resources and memory used by the tenant where the application job is running, as well as the maximum resources and memory that can be requested. This data can be considered to have continuous characteristics.

[0067] The label of each application job is used to indicate the execution engine that each application job is suitable for. For example, we can label applications suitable for Spark execution as positive samples, and applications suitable for MapReduce execution as negative samples.

[0068] For example, when labeling an application, if an application job reports an error when executed with Spark, then this application job can be labeled as a negative sample label; if the efficiency of an application job executed with Spark is more than three times that of executing with MapReduce, then this application job can be labeled as a positive sample label, otherwise it can be labeled as a negative sample label.

[0069] In step 320 , the computer performs model training based on the acquired data to obtain an execution engine prediction model.

[0070] After obtaining the above data, the computer can perform model training based on the obtained data to obtain a better execution engine prediction model.

[0071] In one possible implementation, a computer may input characteristic data of two preset execution engines, characteristic data of multiple application jobs, characteristic data of resources used by these multiple application jobs, and characteristic data of resources provided by each tenant of at least one tenant running these multiple applications into an execution engine prediction model to be trained; the computer may process these data through the execution engine prediction model to be trained to obtain a target prediction probability; and the computer may train the execution engine prediction model to be trained based on the label of each application job and the obtained target prediction probability corresponding to each application job.

[0072] For example, the computer can input the above-collected Spark and MapReduce feature data, as well as the feature data of the above-mentioned multiple application jobs, the feature data of the resources used by the multiple application jobs, and the feature data of the resources provided by each tenant of at least one tenant running the multiple applications into the execution engine prediction model to be trained, and input the target probability corresponding to each application job calculated by the label of each application job and the execution engine prediction model to be trained into the loss function, and then use the adaptive momentum (Adam) gradient descent method through neural network back propagation to continuously adjust the model parameters and update the model parameters to achieve the purpose of minimizing the loss, that is, to make the execution engine determined by the model closer and closer to the labeled execution engine, so as to obtain more appropriate model parameters, and then obtain the final execution engine prediction model.

[0073] Optionally, the computer processes these data through the execution engine prediction model to be trained to obtain a target prediction probability, which may include: preprocessing the characteristic data of two preset execution engines, the characteristic data of multiple application jobs, the characteristic data of resources used by these multiple application jobs, and the characteristic data of resources provided by each tenant in at least one tenant running these multiple applications to obtain preprocessed data; the computer then obtains a prediction value based on the preprocessed data; and the computer then obtains a target prediction probability based on the prediction value.

[0074] From the detailed description of the collected data during the data collection process above, we can know that the collected data includes data with continuous characteristics and data with discrete characteristics. Therefore, the data with continuous characteristics and the data with discrete characteristics can be preprocessed separately.

[0075] For example, Z-score normalization can be used to process data with continuous features to obtain continuous feature vectors; one-hot encoding can be used to process data with discrete features to obtain discrete feature vectors.

[0076] For data with continuous features, Z-score normalization can be used for preprocessing. Specifically, the values ​​of the continuous data are uniformly quantified and converted to a state with a mean of 0 and a variance of 1. The quantified feature data follows a standard normal distribution. Because different feature data often have different dimensions and dimensional units, failure to uniformly quantify these data can affect the results of data analysis. Therefore, to minimize these effects, Z-score normalization is used to normalize the collected continuous data. This allows the various feature data to be on the same order of magnitude, making them suitable for comprehensive comparative evaluation.

[0077] Data after Z-score normalization can be saved in the format of "feature number: feature preprocessed value". For example, the first data point with a continuous feature can be processed as "1:0.5"; the second data point with a continuous feature can be processed as "2:0.004975"; and the third data point with a continuous feature can be processed as "3:0.11". The preprocessed values ​​of these data with continuous features can also be used to form a continuous feature vector. The "feature preprocessed values" of the features of the data after Z-score standardization can be taken to form a continuous feature vector, for example, [0.5, 0.004975, 0.11, ...]. For the sake of convenience, this continuous feature vector is recorded as feat_vals. The first value in this continuous feature vector corresponds to the value of the first data with continuous features after processing, the second value in the continuous feature vector corresponds to the value of the second data with continuous features after processing, the third value in the continuous feature vector corresponds to the value of the third data with continuous features after processing, and so on. I will not go into details here.

[0078] For data with discrete features, one-hot encoding can be used for preprocessing. That is, an N-bit state register is used to encode N states (N ≥ 1, N is an integer). Each state has its own independent register bit, and only one of the N bits is valid.

[0079] One-hot encoded data can be saved in the format of "Feature Number: 1". For example, if there are 13 continuous features, the first discrete feature is numbered starting at 14. For example, if the first discrete feature is "0, 1, 0" after one-hot encoding, it can be saved as "15:1"; if the second discrete feature is "0, 0, 0, 1, 0" after one-hot encoding, it can be saved as "20:1"; and if the third discrete feature is "1, 0, 0" after one-hot encoding, it can be saved as "22:1". The preprocessed values ​​of these discrete features can also be constructed into a continuous feature vector. This can be constructed by taking the "feature number" of the one-hot encoded data, for example, [15, 20, 22, ...]. For ease of description, this discrete feature vector is denoted as feat_ids.

[0080] Optionally, the computer obtains a predicted value based on preprocessed data, which may include: the computer performs dimensionality reduction processing on the discrete feature vector to obtain a dense vector; the computer calculates the inner product of the dense vector and the continuous feature vector to obtain a first predicted value; and / or the computer inputs the first predicted value into the MLP layer to obtain a second predicted value.

[0081] For example, the computer can input feat_vals into Figure 4 The embedding layer of the deep neural network shown in the figure reduces the dimensionality of feat_vals to obtain a dense vector. The computer can then calculate the inner product of this dense vector and feat_ids to obtain the first predicted value, which is recorded as y_v for ease of description.

[0082] For example, y_v may be input into the MLP to obtain a second predicted value. For ease of description, the second predicted value is recorded as y_deep.

[0083] In this implementation, the predicted value obtained by the computer based on the preprocessed data may be y_v, or y_deep, or the sum of y_v and y_deep may be obtained (y_v+y_deep), which is not limited in this application.

[0084] Optionally, the computer obtains the target prediction probability based on the prediction value, which may include: inputting the prediction value into a conversion function to obtain the target prediction probability.

[0085] The conversion function can be used to convert the predicted value into a value between 0 and 1, thereby obtaining the target prediction probability. For example, the conversion function can be a tf.sigmoid function.

[0086] In combination with the predicted value obtained above, the predicted value is input into the conversion function. For example, y_v can be input into the tf.sigmoid function, or y_deep can be input into the tf.sigmoid function, or (y_v+y_deep) can be input into the tf.sigmoid function.

[0087] It should be understood that the sum of the probability that an application job is suitable for execution by Spark and the probability that it is suitable for execution by MapReduce is 1. Therefore, in the specific implementation process, according to different designs of the parameters of the execution engine prediction model, the application job with a target prediction probability greater than or equal to 0.5 can be determined as suitable for execution by Spark, and the application job with a target prediction probability less than 0.5 can be determined as suitable for execution by MapReduce; or, the application job with a target prediction probability greater than or equal to 0.5 can be determined as suitable for execution by MapReduce, and the application job with a target prediction probability less than 0.5 can be determined as suitable for execution by Spark. This application does not limit this.

[0088] After obtaining the target prediction probability, the cross-entropy loss function can be used to define the loss function. The target probability corresponding to each application job calculated by the label of each application job and the execution engine prediction model to be trained can be input into the loss function. Then, the Adam gradient descent method is used through neural network backpropagation to continuously adjust the model parameters and update the model parameters to achieve the purpose of minimizing the loss. That is, the execution engine determined by the model is closer and closer to the labeled execution engine, thereby obtaining more appropriate model parameters and finally obtaining the final execution engine prediction model.

[0089] Based on the above scheme, the model is trained by comprehensively considering the characteristics of the application job, the characteristics of the resources used by the application job, the characteristics of the resources provided by each tenant in at least one tenant running the application, and the characteristics of the two preset execution engines, so that a better execution engine prediction model that integrates the above-mentioned characteristic factors can be obtained. The execution engine prediction model can then be used to determine the appropriate execution engine for each application job to be executed, which is conducive to improving the execution efficiency of the application job.

[0090] Furthermore, after the execution engine prediction model is put into use, its accuracy may decrease over time. To ensure model accuracy, a self-learning and self-updating mechanism, as well as an automatic comparison and release mechanism, can be introduced. For example, one possible implementation involves a computer periodically acquiring the data; training the execution engine prediction model based on the data acquired during the most recent period to obtain an updated execution engine prediction model.

[0091] The period can be one month, three months, etc., and this application does not limit this.

[0092] For example, if the training cycle is one month, model self-learning and self-update can be automatically triggered each month to obtain the last few months' worth of data and retrain the model. After model training, the performance of the new and old models on the latest label set can be compared. If the new model performs better, the new model is released; if the old model performs better, the model can be left unchanged.

[0093] In addition to automatically triggering the stock data every month, the incremental data generated on that day can be annotated every day. Then, the latest model file as of yesterday is downloaded and used as the pre-trained model. The incremental data generated on that day is divided into incremental training samples and test samples in a ratio of 8:2 for incremental model training. After several rounds of training, if it is found that the latest model performs better in the test samples, the model file is updated. In this way, the latest model can be directly applied to the next day's prediction to ensure the accuracy of the model.

[0094] In this way, the model parameters can be updated and adjusted based on the feature data in the most recent cycle to ensure accuracy. After the execution engine prediction model is put into use, the model can be iterated through data-driven means to achieve the goal of one-time online and lifelong learning.

[0095] The following describes a method for determining an execution engine with reference to the accompanying drawings.

[0096] Figure 5 This is a flow chart of a method for determining an execution engine provided by an embodiment of the present application. Figure 5 As shown, the method 500 includes steps 510 to 530. Each step in the method 500 is described in detail below.

[0097] In step 510 , the computer obtains an application job of an execution engine to be determined.

[0098] The computer may obtain an application job for which an execution engine is to be determined, so as to determine a suitable execution engine for the application job according to characteristics of the application job.

[0099] In step 520 , the computer inputs the characteristic data of the application job into an execution engine prediction model. The execution engine prediction model is used to calculate the probability that the application job is applicable to one of two preset execution engines based on the characteristic data of the application job.

[0100] Through method 300, a better execution engine prediction model has been trained. Therefore, when the computer obtains an application job to be executed, the feature data of this application job can be extracted. The data included in the feature data of this application job can refer to the relevant content of the data collection part above. After constructing the feature data of the application job, the feature data of the application job is input into the execution engine prediction model. The execution engine prediction model calculates a probability based on the feature data of this application job. This probability can indicate which of the two preset execution engines is suitable for executing this application job.

[0101] In step 530, the computer sets a target execution engine for the application job based on the probability.

[0102] After the execution engine prediction model outputs a probability, the probability can be used to determine which of the two preset execution engines is most suitable for executing the application job. For example, if the probability determines that Spark is suitable for executing the application job, the computer can automatically generate a "set hive.execution.engine = spark;" statement to set Spark as the execution engine for the application job. For another example, if the probability determines that MapReduce is suitable for executing the application job, the computer can automatically generate a "set hive.execution.engine = mr;" statement to set MapReduce as the execution engine for the application job.

[0103] Based on the above scheme, when an application job needs to be executed, the execution engine prediction model can be used to calculate the probability that the application job is suitable for one of the two preset execution engines, and then a suitable target execution engine can be set for the application job based on the calculated probability, thereby automatically setting a suitable execution engine for the application job, which is conducive to improving the execution efficiency of the application job.

[0104] Furthermore, the need for manual configuration and monitoring of specific execution engines is eliminated, reducing development workload. Automatically configuring the execution engine for application jobs improves the efficiency of collecting massive amounts of data and analyzing customer behavior on the application side, enhancing the timeliness of data updates on the big data platform. This also prevents occasional interruptions in Spark application jobs caused by resource constraints across the entire cluster or for individual tenants, improving the stability of production systems and ultimately increasing cluster resource utilization.

[0105] Figure 6 This is a schematic block diagram of a determination device of an execution engine provided in an embodiment of the present application.

[0106] like Figure 6 As shown, the execution engine determination device 600 may include: an acquisition module 610, an input module 620, and a setting module 630. The execution engine determination device 600 may be used to implement the functions of the computer in the above-mentioned method 500, wherein the acquisition module 610 may be used to acquire an application job for which an execution engine is to be determined; the input module 620 may be used to input feature data of the application job into an execution engine prediction model, the execution engine prediction model being used to calculate the probability that the application job is applicable to one of two preset execution engines based on the feature data of the application job; and the setting module 630 may be used to set a target execution engine for the application job based on the probability, the target execution engine being one of the two preset execution engines.

[0107] Figure 7 This is a schematic block diagram of a model training device provided in an embodiment of the present application.

[0108] like Figure 7 As shown, the model training device 700 may include: an acquisition module 710 and a training module 720. The model training device 700 may be used to implement the functions of the computer in the above-mentioned method 300, wherein the acquisition module 710 may be used to acquire data, the data including feature data of two preset execution engines, feature data of multiple application jobs, labels of each application job in the multiple application jobs, feature data of resources used by the multiple application jobs, and feature data of resources provided by each tenant in at least one tenant running the multiple applications, the label of each application job is used to indicate the execution engine applicable to the each application job; the training module 720 may be used to perform model training based on the data to obtain an execution engine prediction model, the execution engine prediction model is used to predict the execution engine applicable to the application job.

[0109] Optionally, the training module 720 can also be specifically used to input the characteristic data of the preset two execution engines, the characteristic data of multiple application jobs, the characteristic data of the resources used by the multiple application jobs, and the characteristic data of the resources provided by each tenant in at least one tenant running the multiple applications into the execution engine prediction model to be trained; the characteristic data of the preset two execution engines, the characteristic data of multiple application jobs, the characteristic data of the resources used by the multiple application jobs, and the characteristic data of the resources provided by each tenant in at least one tenant running the multiple applications are processed by the execution engine prediction model to be trained to obtain a target prediction probability; and the execution engine prediction model to be trained is trained based on the label of each application job and the target prediction probability.

[0110] Optionally, the training module 720 can also be used to preprocess the characteristic data of the two preset execution engines, the characteristic data of multiple application jobs, the characteristic data of the resources used by the multiple application jobs, and the characteristic data of the resources provided by each tenant in at least one tenant running the multiple applications to obtain preprocessed data; obtain a prediction value based on the preprocessed data; and obtain the target prediction probability based on the prediction value.

[0111] Optionally, the training module 720 can also be used to process data with continuous features using Z-score normalization to obtain continuous feature vectors; and to process data with discrete features using one-hot encoding to obtain discrete feature vectors.

[0112] Optionally, the training module 720 can also be used to perform dimensionality reduction processing on the discrete feature vector to obtain a dense vector; calculate the inner product of the dense vector and the continuous feature vector to obtain a first prediction value; and / or input the first prediction value into the MLP to obtain a second prediction value.

[0113] Optionally, the training module 720 may be further configured to input the predicted value into a conversion function to obtain the target predicted probability, wherein the conversion function is configured to convert the predicted value into a value between 0 and 1.

[0114] Optionally, the acquisition module 710 may also be used to periodically acquire the data; the training module 720 may also be used to train the execution engine prediction model based on the data acquired in the most recent period to obtain an updated execution engine prediction model.

[0115] Figure 8 This is a schematic block diagram of another device provided in an embodiment of the present application.

[0116] The device 800 can be used to implement the functions of the computer in the above-mentioned method 300 and / or method 500. The device can be a chip system. In the embodiment of the present application, the chip system can be composed of a chip, or can include a chip and other discrete devices.

[0117] like Figure 8 As shown, the apparatus 800 may include at least one processor 810 for implementing the functions of the computer in the method 300 and / or the method 500 provided in the embodiments of the present application.

[0118] Exemplarily, when the device 800 is used to implement the method 300 provided in an embodiment of the present application, the processor 810 can be used to obtain data, the data including characteristic data of two preset execution engines, characteristic data of multiple application jobs, a label of each application job in the multiple application jobs, characteristic data of resources used by the multiple application jobs, and characteristic data of resources provided by each tenant in at least one tenant running the multiple applications, wherein the label of each application job is used to indicate the execution engine applicable to each application job; and perform model training based on the data to obtain an execution engine prediction model, wherein the execution engine prediction model is used to predict the execution engine applicable to the application job. For details, please refer to the detailed description in the method example, which is not repeated here.

[0119] Exemplarily, when the apparatus 800 is used to implement the method 500 provided in an embodiment of the present application, the processor 810 may be configured to obtain an application job for a to-be-determined execution engine; input characteristic data of the application job into an execution engine prediction model, the execution engine prediction model being configured to calculate, based on the characteristic data of the application job, a probability that the application job is applicable to one of two preset execution engines; and, based on the probability, set a target execution engine for the application job, the target execution engine being one of the two preset execution engines. For details, please refer to the detailed description in the method example and will not be elaborated upon here.

[0120] The device 800 may also include at least one memory 820, which may be used to store multiple token codes. The memory 820 is coupled to the processor 810. In the embodiments of the present application, coupling refers to an indirect coupling or communication connection between devices, units, or modules, which may be electrical, mechanical, or other forms, and is used for information exchange between the devices, units, or modules. The processor 810 may operate in conjunction with the memory 820. The processor 810 may execute program instructions stored in the memory 820. At least one of the at least one memory may be included in the processor.

[0121] The apparatus 800 may also include a communication interface 830 for communicating with other devices via a transmission medium, thereby enabling the apparatus 800 to communicate with other devices. The communication interface 830 may be, for example, a transceiver, an interface, a bus, a circuit, or a device capable of performing transceiver functions. The processor 810 may utilize the communication interface 830 to transmit and receive data and / or information, and to implement Figure 3 The method 300 executed by the computer in the corresponding embodiment, and / or, Figure 5 The method 500 executed by a computer in the corresponding embodiment.

[0122] The specific connection medium between the processor 810, the memory 820 and the communication interface 830 is not limited in the embodiment of the present application. Figure 8 The processor 810, the memory 820 and the communication interface 830 are connected via a bus 840. The bus 840 is connected to the Figure 8 The connections between the other components are shown in bold lines, which are only for illustration and are not intended to be limiting. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, Figure 8 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0123] The present application also provides a chip system, which includes at least one processor for implementing the above Figure 3 The functions involved in the computer-implemented method described in the illustrated embodiments, and / or, Figure 5 The functions involved in the computer-implemented method described in the illustrated embodiments.

[0124] In one possible design, the chip system further includes a memory, which is used to store program instructions and data, and the memory is located inside or outside the processor.

[0125] The chip system can be composed of chips, or can include chips and other discrete devices.

[0126] The present application also provides a computer program product, which includes: a computer program (also referred to as code or instruction), which, when executed, causes a computer to execute the following Figure 3 and / or Figure 5 The method of the embodiment shown.

[0127] The present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program (also referred to as code or instruction). When the computer program is executed, the computer executes the following Figure 3 and / or Figure 5The method of the embodiment shown.

[0128] It should be noted that the execution engine determination method, model training method and device provided in the embodiments of the present application can be applied to the big data field, and can also be applied to any field other than the big data field, such as the artificial intelligence field, and this application does not limit this.

[0129] It should be understood that the processor in the embodiments of the present application can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above-mentioned method embodiment can be completed by hardware integrated logic circuits in the processor or by software instructions. The above-mentioned processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of the present application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above-mentioned method.

[0130] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0131] The terms "unit," "module," and the like used in this specification may be used to refer to a computer-related entity, hardware, firmware, a combination of hardware and software, software, or software in execution.

[0132] Those skilled in the art will appreciate that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented using electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application. In the several embodiments provided in this application, it should be understood that the disclosed devices, equipment, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not performed. In addition, the coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of devices or units, and can be electrical, mechanical, or other forms.

[0133] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules may be selected to achieve the purpose of the present embodiment according to actual needs.

[0134] In addition, the functional modules in the various embodiments of the present application may be integrated into one processing module, or each module may exist physically separately, or two or more units may be integrated into one module.

[0135] In the above embodiments, the functions of each functional module can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions (programs). When the computer program instructions (program) are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital versatile disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0136] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.

[0137] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for determining an execution engine, characterized in that: The method comprises: Get the application job of the execution engine to be determined; Inputting the characteristic data of the application job into an execution engine prediction model, the execution engine prediction model is used to calculate the probability that the application job is applicable to one of the two preset execution engines based on the characteristic data of the application job, the execution engine prediction model is obtained by model training based on data, the data including the characteristic data of the two preset execution engines, the characteristic data of multiple application jobs, the label of each application job in the multiple application jobs, the characteristic data of resources used by the multiple application jobs, and the characteristic data of resources provided by each tenant in at least one tenant running the multiple applications, the label of each application job is used to indicate the execution engine to which each application job is applicable; the execution engine prediction model can also be trained and updated based on the data obtained in the most recent cycle; the two execution engines simultaneously start to execute the same batch of query statements; the characteristic data of the application job includes the dependency information of each query statement included in the current application job in this application job; the characteristic data of the resources used by the application job includes the total number of query statements currently running on the entire Hadoop platform where the application is running, the resources and memory that have been allocated on the entire Hadoop platform, and the amount of resources and memory remaining; A target execution engine is set for the application job based on the probability, where the target execution engine is one of the two preset execution engines.

2. A model training method, characterized in that: The method comprises: Acquire data, the data including characteristic data of two preset execution engines, characteristic data of multiple application jobs, a label of each application job in the multiple application jobs, characteristic data of resources used by the multiple application jobs, and characteristic data of resources provided by each tenant of at least one tenant running the multiple applications, wherein the label of each application job is used to indicate the execution engine applicable to each application job; the two execution engines simultaneously start to execute the same batch of query statements; the characteristic data of the application jobs includes dependency information of each query statement included in the current application job in the application job; the characteristic data of the resources used by the application jobs includes the total number of query statements currently running on the entire Hadoop platform on which the application is running, the resources and memory that have been allocated on the entire Hadoop platform, and the amount of resources and memory remaining; Performing model training based on the data to obtain an execution engine prediction model, wherein the execution engine prediction model is used to predict an execution engine suitable for an application job; periodically acquiring the data; The execution engine prediction model is trained based on the data acquired in the most recent cycle to obtain an updated execution engine prediction model.

3. The method according to claim 2, wherein The performing model training based on the data includes: Inputting the feature data of the two preset execution engines, the feature data of the multiple application jobs, the feature data of resources used by the multiple application jobs, and the feature data of resources provided by each tenant of the at least one tenant running the multiple applications into the execution engine prediction model to be trained; Processing, by means of the execution engine prediction model to be trained, the feature data of the two preset execution engines, the feature data of the multiple application jobs, the feature data of resources used by the multiple application jobs, and the feature data of resources provided by each tenant of the at least one tenant running the multiple applications, to obtain a target prediction probability; The execution engine prediction model to be trained is trained based on the label of each application job and the target prediction probability.

4. The method according to claim 3, wherein The step of processing the feature data of the two preset execution engines, the feature data of the multiple application jobs, the feature data of resources used by the multiple application jobs, and the feature data of resources provided by each tenant of the at least one tenant running the multiple applications by the execution engine prediction model to obtain a target prediction probability includes: preprocessing the characteristic data of the two preset execution engines, the characteristic data of the multiple application jobs, the characteristic data of resources used by the multiple application jobs, and the characteristic data of resources provided by each tenant of the at least one tenant running the multiple applications to obtain preprocessed data; Obtaining a predicted value based on the preprocessed data; The target prediction probability is obtained based on the prediction value.

5. The method according to claim 4, wherein The preprocessing of the characteristic data of the two preset execution engines, the characteristic data of the multiple application jobs, the characteristic data of resources used by the multiple application jobs, and the characteristic data of resources provided by each tenant of the at least one tenant running the multiple applications includes: Zero-value Z-score standardization is used to process data with continuous features to obtain continuous feature vectors; One-hot encoding is used to process data with discrete features to obtain discrete feature vectors.

6. The method according to claim 5, wherein The method of obtaining a predicted value based on the preprocessed data includes: Performing dimensionality reduction processing on the discrete feature vector to obtain a dense vector; Calculating the inner product of the dense vector and the continuous feature vector to obtain a first prediction value; and / or The first prediction value is input into a multi-layer perceptron MLP to obtain a second prediction value.

7. The method according to claim 4, wherein Obtaining the target prediction probability based on the prediction value includes: The predicted value is input into a conversion function to obtain the target predicted probability, and the conversion function is used to convert the predicted value into a value between 0 and 1.

8. A device for determining an execution engine, characterized in that: The device comprises: An acquisition module, used to acquire application jobs of a to-be-determined execution engine; An input module is configured to input the characteristic data of the application job into an execution engine prediction model, wherein the execution engine prediction model is configured to calculate the probability that the application job is applicable to one of two preset execution engines based on the characteristic data of the application job. The execution engine prediction model is obtained by model training based on data, wherein the data includes characteristic data of the two preset execution engines, characteristic data of multiple application jobs, a label of each application job in the multiple application jobs, characteristic data of resources used by the multiple application jobs, and characteristic data of resources provided by each tenant in at least one tenant running the multiple applications, wherein the label of each application job is configured to indicate the execution engine to which each application job is applicable. The execution engine prediction model can also be trained and updated based on the data acquired in the most recent cycle. The two execution engines simultaneously start to execute the same batch of query statements. The characteristic data of the application job includes dependency information of each query statement included in the current application job in the application job. The characteristic data of the resources used by the application job includes the total number of query statements currently running on the entire Hadoop platform on which the application is running, the resources and memory that have been allocated on the entire Hadoop platform, and the amount of resources and memory remaining. A setting module is configured to set a target execution engine for the application job based on the probability, where the target execution engine is one of the two preset execution engines.

9. A device for determining an execution engine, characterized in that: A processor is included that executes computer-implemented instructions to implement the method of claim 1.

10. A model training device, characterized in that: The device comprises: An acquisition module is configured to acquire data, the data including characteristic data of two preset execution engines, characteristic data of multiple application jobs, a label of each of the multiple application jobs, characteristic data of resources used by the multiple application jobs, and characteristic data of resources provided by each tenant of at least one tenant running the multiple applications, wherein the label of each application job is used to indicate the execution engine applicable to each application job; the two execution engines are simultaneously started to execute the same batch of query statements; the characteristic data of the application jobs includes dependency information of each query statement included in the current application job within the application job; the characteristic data of the resources used by the application jobs includes the total number of query statements currently running on the entire Hadoop platform on which the application is running, the resources and memory that have been allocated on the entire Hadoop platform, and the amount of resources and memory remaining; a training module, configured to perform model training based on the data to obtain an execution engine prediction model, wherein the execution engine prediction model is used to predict an execution engine suitable for an application job; periodically acquiring the data; The execution engine prediction model is trained based on the data acquired in the most recent cycle to obtain an updated execution engine prediction model.

11. A model training device, characterized in that: comprising a processor that executes computer-implemented instructions to implement the method of any one of claims 2 to 7.

12. A computer-readable storage medium, characterized in that The method comprises a computer program, which, when running on a computer, causes the computer to perform the method according to any one of claims 1 to 7.

13. A computer program product, characterized in that The invention comprises a computer program which, when being executed, causes a computer to execute the method according to any one of claims 1 to 7.

Images