Information processing method, device, equipment and storage medium

Abstract

Embodiments of the present application provide an information processing method, apparatus, device, and storage medium for use in computer technology. This method includes obtaining target features corresponding to a target Spark task; determining parameters of the target Spark task based on the target features and a parameter determination model, wherein the parameter determination model is used to predict the parameters of the Spark task, and the parameter determination model is trained based on historical Spark task operation information. This application enables automatic adjustment of Spark task parameters, reducing labor costs while improving the efficiency of obtaining Spark task parameters.

Description

Technical Field

[0001] The present application relates to big data technology, and in particular to an information processing method, apparatus, device and storage medium. Background Art

[0002] Spark is a fast and versatile computing engine designed for large-scale data processing. Compared to Hadoop, Spark enables in-memory distributed datasets. In addition to providing interactive queries, it can also optimize iterative workloads and support iterative jobs on distributed datasets.

[0003] In actual applications, after a user submits a Spark task, the Spark task will start a corresponding driver process, which needs to apply to the cluster manager for an executor to run the Spark task. The cluster manager will start a certain number of executors on each worker node in the cluster based on the parameters set by the user for the Spark task. Each executor occupies a certain amount of memory and processor cores.

[0004] In the process of implementing this application, the inventors found that there are at least the following problems in the prior art: the parameters of Spark tasks are determined one by one by relevant personnel based on historical experience, which is inefficient and consumes a lot of manpower. Summary of the Invention

[0005] The embodiments of the present application provide an information processing method, apparatus, device, and storage medium, which can efficiently determine the parameters of Spark tasks and save manpower.

[0006] In a first aspect, an embodiment of the present application provides an information processing method, including:

[0007] Get the target features corresponding to the target Spark task;

[0008] The parameters of the target Spark task are determined based on the target features and parameter determination model. The parameter determination model is used to predict the parameters of the Spark task. The parameter determination model is trained based on the historical operation information of the Spark task.

[0009] In one possible implementation, determining the model based on target features and parameters to determine the parameters of the target Spark task includes:

[0010] Parameterize the target feature and obtain the encoding value of the target feature;

[0011] Input the encoded value into the parameter determination model to obtain the parameters of the target Spark task.

[0012] In one possible implementation, after determining the model based on the characteristic values ​​and parameters and determining the parameters of the target Spark task, the following steps may be further included:

[0013] Set the parameters of the target Spark task; or,

[0014] Adjust the current parameters of the target Spark task to the parameters obtained through the parameter determination model.

[0015] In one possible implementation, the parameter determination model is trained by the following steps:

[0016] Get the historical running information of Spark tasks;

[0017] Extract Spark task feature information from historical running information;

[0018] Iteratively train the initial parameter determination model based on the characteristic information of the Spark task until the calculated evaluation index value meets the preset evaluation conditions;

[0019] Determine that the model corresponding to the evaluation index value when it meets the preset evaluation conditions is the parameter determination model;

[0020] The structure of the initial parameter determination model is the same as that of the parameter determination model.

[0021] In a third aspect, an embodiment of the present application provides a method for training a parameter determination model, including:

[0022] Get the historical running information of Spark tasks;

[0023] Extract Spark task feature information from historical running information;

[0024] Iteratively train the initial parameter determination model based on the characteristic information of the Spark task until the calculated evaluation index value meets the preset evaluation conditions;

[0025] Determine that the model corresponding to the evaluation index value when it meets the preset evaluation conditions is the parameter determination model;

[0026] The initial parameter determination model has the same structure as the parameter determination model, and the parameter determination model is used to predict the parameters of the Spark task.

[0027] In one possible implementation, the feature information includes feature values ​​corresponding to the features, and iteratively training the initial parameter determination model based on the feature information of the Spark task includes:

[0028] determining target features from features included in the feature information;

[0029] Based on the target features and the eigenvalues ​​corresponding to the target features, the initial parameters are iteratively trained to determine the model.

[0030] In one possible implementation, determining the target feature from the features included in the feature information includes:

[0031] Determine the correlation between the features based on the feature values ​​corresponding to the features;

[0032] The features whose relevance meets the preset similarity conditions are determined as target features.

[0033] In a possible implementation, the method further includes: after obtaining the parameter determination model through iterative training, determining the parameters of the parameter determination model by a cross-validation method.

[0034] In a third aspect, an embodiment of the present application provides an information processing device, including:

[0035] The acquisition module is used to obtain the encoding value of the target feature corresponding to the target Spark task;

[0036] The determination module is used to determine the parameter values ​​of the target Spark task according to the encoding value and the parameter determination model. The parameter determination model is used to predict the parameters of the Spark task. The parameter determination model is trained based on the historical operation information of the Spark task.

[0037] In one possible implementation, the determination module is specifically configured to:

[0038] Parameterize the target feature and obtain the encoding value of the target feature;

[0039] Input the encoded value into the parameter determination model to obtain the parameters of the target Spark task.

[0040] In a possible implementation, the determination module is further configured to:

[0041] Set the parameters of the target Spark task; or,

[0042] Adjust the current parameters of the target Spark task to the parameters obtained through the parameter determination model.

[0043] In one possible implementation, the parameter determination model is trained by the following steps:

[0044] Get the historical running information of Spark tasks;

[0045] Extract Spark task feature information from historical running information;

[0046] Iteratively train the initial parameter determination model based on the characteristic information of the Spark task until the calculated evaluation index value meets the preset evaluation conditions;

[0047] Determine that the model corresponding to the evaluation index value when it meets the preset evaluation conditions is the parameter determination model;

[0048] The structure of the initial parameter determination model is the same as that of the parameter determination model.

[0049] In a fourth aspect, an embodiment of the present application provides a training device for a parameter determination model, comprising:

[0050] The acquisition module is used to obtain the historical running information of Spark tasks;

[0051] The extraction module is used to extract the characteristic information of Spark tasks from historical running information;

[0052] The training module is used to iteratively train the initial parameter determination model based on the characteristic information of the Spark task until the calculated evaluation index value meets the preset evaluation conditions;

[0053] A determination module, used to determine that the corresponding model when the evaluation index value meets the preset evaluation conditions is a parameter determination model;

[0054] The initial parameter determination model has the same structure as the parameter determination model, and the parameter determination model is used to predict the parameters of the Spark task.

[0055] In one possible implementation, the feature information includes feature values ​​corresponding to the features, and the training module is specifically configured to:

[0056] determining target features from features included in the feature information;

[0057] Based on the target features and the eigenvalues ​​corresponding to the target features, the initial parameters are iteratively trained to determine the model.

[0058] In one possible implementation, the training module is specifically configured to:

[0059] Determine the correlation between the features based on the feature values ​​corresponding to the features;

[0060] The features whose relevance meets the preset similarity conditions are determined as target features.

[0061] In a possible implementation, the following further includes:

[0062] The validation module is used to determine the parameters of the model using a cross-validation method.

[0063] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method described in any one of the first aspect and / or the second aspect is implemented.

[0064] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program runs on an electronic device, the electronic device executes a method as described in any one of the first aspect and / or the second aspect.

[0065] In a seventh aspect, an embodiment of the present application provides a computer program product, comprising a computer program. When the computer program is run on an electronic device, the electronic device executes the method as described in any one of the first aspect and / or the second aspect.

[0066] The information processing method, apparatus, device, and storage medium provided in the embodiments of the present application acquire target features corresponding to a target Spark task; determine the parameters of the target Spark task based on the target features and a parameter determination model; the parameter determination model is used to predict the parameters of the Spark task; and the parameter determination model is trained based on historical Spark task operation information. This application can automatically adjust Spark task parameters, reducing labor costs while improving the efficiency of obtaining Spark task parameters.

[0067] These and other aspects of the present application will become more apparent from the following description of the embodiment(s). BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0069] Figure 1 A schematic diagram of an application scenario of the information processing method provided in one embodiment of the present application;

[0070] Figure 2 A flowchart of an information processing method provided in one embodiment of the present application;

[0071] Figure 3 A flowchart of an information processing method provided in another embodiment of the present application;

[0072] Figure 4 A flowchart of a method for training a parameter determination model according to an embodiment of the present application;

[0073] Figure 5 A schematic diagram of the structure of an information processing device provided in one embodiment of the present application;

[0074] Figure 6 A schematic diagram of the structure of a training device for a parameter determination model provided in one embodiment of the present application;

[0075] Figure 7 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application.

[0076] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0077] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0078] First, some of the terms involved in the embodiments of this application are explained as follows:

[0079] REST API: It is a set of architectural rules, standards or guidelines on how to build a web application API. REST API follows the architectural style of API principles. REST is designed specifically for web applications with the aim of reducing development complexity and improving system scalability.

[0080] Spark: An open-source parallel computing framework developed by the UC Berkeley AMP lab, it is primarily used for batch processing and near-real-time computation of large data volumes. Spark tasks are user-written business function codes that are executed in Spark.

[0081] One-hot encoding: One-hot encoding, also known as single-bit encoding, primarily uses an N-bit state register to encode N states. Each state is represented by its own register bit, and only one bit is active at any given time. One-hot encoding represents categorical variables as binary vectors. Categorical values ​​are first mapped to integer values. Each integer value is then represented as a binary vector where all values ​​are zero except for the integer index, which is marked as 1.

[0082] Considering that the Spark task parameters of the current data warehouse need to be manually adjusted, each adjustment requires manual stress testing of the Spark task to obtain the corresponding Spark task parameters. This process is inefficient and requires a lot of manpower.

[0083] To address the above issues, the present application provides an information processing method, apparatus, device, and storage medium, which use machine learning to automatically adjust task parameters according to the task's running status, thereby reducing labor costs while improving the efficiency of obtaining Spark task parameters.

[0084] For example, Figure 1 Schematic diagram of an application scenario of the information processing method provided in one embodiment of the present application. Figure 1 As shown, the application scenario may include: a client 101 and a server 102 .

[0085] In the embodiments of the present application, client 101 can be any type of client, for example, a user equipment (UE) for machine-type communication. Client 101 can also be referred to as user equipment (UE), mobile station (MS), mobile terminal (mobile terminal), terminal, etc. For example, client 101 can be a desktop computer, laptop, personal digital assistant (PDA), smartphone, tablet computer, automotive product, wearable device, etc. This scenario uses a desktop computer as an example.

[0086] Server 102 may include a business point that provides processing, database, and communication facilities. Server 102 may be a monolithic server or a distributed server or server cluster across multiple computers or computer data centers. Server 102 may be of various types, such as, but not limited to, a web server, a messaging server, an application server, an interactive server, a database server, or a proxy server.

[0087] The client 101 can communicate with the server 102 via a wireless or wired network to send data to the server 102. The wireless network can be a 2G, 3G, 4G, or 5G communication network, or a wireless local area network, which is not limited here.

[0088] In actual applications, the above-mentioned client 101 can be deployed on the server 10 in the Spark environment. When the client 101 obtains data, the server 102 starts the Spark computing task, so that Spark can calculate the obtained data according to the task parameters and related auxiliary data in Spark to generate calculation results.

[0089] In an embodiment of the present application, a parameter determination model is deployed in the server 102. When a Spark computing task is required, the server 102 first determines the task parameters of the Spark task based on the parameter determination model, and then calculates the obtained data based on the determined task parameters and related auxiliary data to generate a calculation result.

[0090] It should be noted that the attached Figure 1 This is only a schematic diagram of an application scenario provided by the embodiment of the present application. Figure 1 The number of devices and equipment included in the Figure 1 The positional relationship between the devices in the Figure 1 The application scenario shown may also include a data storage device, which may be an external memory relative to the server 102 or an internal memory integrated in the server 102. In addition, the server 102 may be an independent server or a service cluster.

[0091] The technical solution of the present application is described in detail below through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0092] Figure 2 This is a flow chart of an information processing method provided by an embodiment of the present application. This embodiment of the present application provides an information processing method, which is applied to an information processing device, and the device can be implemented in software and / or hardware. Figure 1 In the scenario shown, the information processing device can be integrated into Figure 1 In the server shown, for example, the information processing device is a chip or circuit in the server; or, the information processing device is Figure 1 Next, an example is given using the server as the execution subject.

[0093] like Figure 2 As shown, the information processing method includes the following steps:

[0094] S201. Obtain target features corresponding to the target Spark task.

[0095] In practical applications, different Spark tasks correspond to different target features. The target features can be determined based on the correspondence between the Spark tasks and the target features. In other words, the target features are pre-bound to the target Spark tasks. The correspondence between the Spark tasks and the target features can be set based on the actual requirements of the target Spark tasks and is not specifically limited in this embodiment.

[0096] Specifically, the target feature may include at least one of the following types: task appinfo data, job data, stage data, executor data, experiment (environment) data, tasksummary data, etc.

[0097] The task appinfo data may specifically include at least one of the following features: application ID, task ID, task instance ID, cluster, marketplace, user, queue, start time, end time, execution duration, etc.

[0098] Specifically, the job data may include at least one of the following features: application ID, job ID, submission time, and completion time;

[0099] The stage data may include at least one of the following features: application ID, attempt ID, job ID, stage ID, stage attempt number, number of tasks, input bytes, input records, output bytes, output records, shuffle read bytes, shuffle read records, shuffle write bytes, shuffle records, spilled memory, spilled disk, etc.

[0100] Executor data may specifically include at least one of the following features: application ID, attempt ID, execution ID, JVM heap memory, heap execution memory, off-heap execution memory, major gc count, major gc time, minor gc count, minor gc time, etc.;

[0101] The environment data can specifically include at least one of the following features: application ID, number of executor instances, number of executor cores, executor memory, number of partitions, etc.

[0102] The tasksummary data may specifically include at least one of the following features: application ID, jobID, stageID, number of bytes read, number of records read, shuffle read bytes, shuffle read records, shuffle write bytes, shuffle records, etc.

[0103] S202: Determine a model based on target features and parameters, and determine parameters of the target Spark task.

[0104] The parameter determination model is used to predict the parameters of the Spark task, and the parameter determination model is trained based on the historical running information of the Spark task.

[0105] In practical applications, the embodiment of the present application does not specifically limit the type of parameter determination model. On the one hand, the parameter determination model can be an XGBOOST model, and on the other hand, the parameter determination model can also be a random forest model and a decision tree model.

[0106] Exemplarily, the embodiment of the present application uses the XGBOOST model as a backbone network to illustrate. By inputting the target features into the XGBOOST model for processing, the parameters of the target Spark task are determined. As for the specific method of obtaining the parameters of the target Spark task, please refer to the subsequent embodiments.

[0107] The information processing method provided in the embodiments of the present application obtains target features corresponding to a target Spark task; determines the parameters of the target Spark task based on the target features and a parameter determination model. The parameter determination model is used to predict the parameters of the Spark task, and the parameter determination model is trained based on historical operation information of the Spark task. This application can automatically adjust Spark task parameters, reducing labor costs while improving the efficiency of obtaining Spark task parameters.

[0108] Based on the above embodiment, further, S202, determining a model according to target features and parameters to determine parameters of a target Spark task, may specifically include the following steps:

[0109] (1) Parameterize the target feature and obtain the encoding value of the target feature.

[0110] In machine learning algorithms, these target features need to be calculated and classified. However, these target features are discrete and disordered, not continuous, and parameter determination models cannot be directly applied to them. Therefore, when determining the parameters of the target Spark task based on the target features and parameter determination model, the target features need to be parameterized.

[0111] It should be noted that the embodiments of the present application do not specifically limit the scheme for parameterizing the target features. For example, the target features can be parameterized using a one-hot encoding method to obtain the encoding value of the target features.

[0112] In practical applications, the code value of the target feature can be determined based on the correspondence between the target feature and the code value. Specifically, the correspondence between the target feature and the code value can be obtained by the following method:

[0113] First, the feature values ​​are mapped to integer values. Then, each integer value is represented as a binary vector. All values ​​except the integer index are zero, and the integer index is marked as 1. The following example illustrates the process of parameterizing the target feature using the one-hot encoding method:

[0114] Taking the target feature as the executor memory in the executor data as an example, for example, when the executor memory is classified into 4G, 6G, 8G, 10G, and 12G, the corresponding index value of each executor memory can be determined according to the corresponding relationship table between some executor memories and their indexes shown in Table 1 below. Then, the encoding value of the executor memory can be determined according to the index value of the executor memory:

[0115] Table 1 Correspondence between executor memory and its index

[0116]

[0117] It can be concluded from Table 1 above that when the Executor memory is 4G, its corresponding encoding value is: 10000; when the Executor memory is 6G, its corresponding encoding value is: 01000; when the Executor memory is 8G, its corresponding encoding value is: 00100 When the Executor memory is 10G, its corresponding encoding value is: 00010 When the Executor memory is 12G, its corresponding encoding value is: 00001.

[0118] It is understandable that the scheme for determining the coding values ​​corresponding to other target features is similar to the above scheme and will not be described in detail here.

[0119] (2) Input the encoded value into the parameter determination model to obtain the parameters of the target Spark task.

[0120] Specifically, the encoded value is input into the parameter determination model to obtain the parameters of the target Spark task. It should be noted that the specific solution for the parameter determination model to determine the parameters of the target Spark task based on the encoded value can be referred to the solution in the prior art and will not be repeated here.

[0121] Based on the above embodiment, further, after determining the model based on the target characteristics and parameters and determining the parameters of the target Spark task in S202, it is necessary to adjust the target Spark task based on the determined parameters of the target Spark task so that the corresponding business is executed through the adjusted target Spark task. Specifically, the target Spark task can be adjusted in the following manner:

[0122] On the one hand, you can adjust the target Spark task by setting its parameters;

[0123] It is understandable that if the target Spark task is used for the first time, that is, if the parameters of the target Spark task have not been set yet, the determined parameters of the target Spark task are directly set as the parameters of the target Spark task.

[0124] On the other hand, the current parameters of the target Spark task may be adjusted to the parameters obtained by the parameter determination model to adjust the target Spark task.

[0125] In actual applications, if the target Spark task is not used for the first time, that is, the target Spark task has been set with parameters, then the current parameters of the target Spark task are adjusted to the parameters obtained by the parameter determination model.

[0126] In some embodiments, before determining the parameters of the target Spark task based on the target features and parameter determination model, it is also necessary to train the initial parameter determination model to obtain the parameter determination model. Figure 3 A detailed description of the training process of the parameter determination model:

[0127] Figure 3 This is a flow chart of an information processing method provided by another embodiment of the present application. Figure 3 As shown, the information processing method provided in the embodiment of the present application may include the following steps:

[0128] S301. Obtain historical running information of Spark tasks.

[0129] It should be noted that the embodiment of the present application does not specifically limit the method for obtaining the historical operation information of the Spark task. On the one hand, it can be directly obtained from the storage medium of the server. For example, during the operation of the Spark task, the operation information of the Spark task will be stored in the storage medium. When the parameter determination model is trained, the historical operation information of the Spark task is directly obtained from the storage medium.

[0130] Alternatively, you can access Spark task historical execution information through the Rest API. This method can be implemented using existing techniques and will not be further elaborated here. This method, which uses the Rest API to access Spark task historical execution information, allows for easy monitoring of task duration, stages, and other tasks. Furthermore, it can be used in conjunction with a time series database to monitor individual tasks in the cluster. This allows for quick and accurate access to Spark task historical execution information, further improving model training efficiency.

[0131] In actual applications, corresponding to the target features, the historical running information of the Spark task may include at least one of the following: task appinfo data, job data, stage data, executor data, experiment (environment) data, tasksummary data, etc.

[0132] The task appinfo data may include at least one of the following: application ID, task ID, task instance ID, cluster, marketplace, user, queue, start time, end time, execution duration, etc.

[0133] Specifically, the job data includes at least one of the following data: application ID, job ID, submission time, and completion time;

[0134] Stage data may include at least one of the following: application ID, attempt ID, jobID, stageID, stage attempt number, number of tasks, input bytes, input records, output bytes, output records, shuffle read bytes, shuffle read records, shuffle write bytes, shuffle records, spilled memory, spilled disk, etc.

[0135] Executor data may include at least one of the following: application ID, attempt ID, execution ID, JVM heap memory, heap execution memory, off-heap execution memory, major gc count, major gc time, minor gc count, minor gc time, etc.

[0136] Environment data can include at least one of the following: application ID, number of executor instances, number of executor cores, executor memory, number of partitions, etc.

[0137] Task summary data may include at least one of the following: application ID, jobID, stageID, number of read bytes, number of read records, shuffle read bytes, shuffle read records, shuffle write bytes, shuffle records, etc.

[0138] S302: Extract characteristic information of the Spark task from historical operation information.

[0139] In practical applications, the embodiments of the present application do not specifically limit the method for obtaining feature information and the format of the feature information. For example, the above historical operation information can be processed into a wide table with the application ID as the primary key to obtain the feature information of the Spark task.

[0140] Specifically, the wide table with application ID as the primary key includes at least one of the following data: application ID, maximum running JVM heap memory, maximum running heap execution memory, maximum running off-heap execution memory, maximum running major GC number, maximum running major GC time, maximum running minor GC number, maximum running minor GC time, average running JVM heap memory, average running heap execution memory, average running off-heap execution memory, average running major GC number, average running major GC time, average running minor GC number, average running minor GC time, number of executor instances, number of executor cores, executor memory, number of partitions, etc.

[0141] It should be noted that the method of processing the above historical operation information into a wide table with the application ID as the primary key can refer to the existing technology and will not be repeated here.

[0142] S303: Iteratively train the initial parameter determination model based on the characteristic information of the Spark task until the calculated evaluation index value meets the preset evaluation condition.

[0143] The initial parameter determination model and the parameter determination model have the same structure. For example, the initial parameter determination model and the parameter determination model are both one of the XGBOOST model, the random forest model, and the decision tree model. It is understandable that the embodiment of the present application uses the XGBOOST model as the backbone network for illustration.

[0144] The specific solution of S303 is described in detail below in conjunction with steps S3031 to S3032:

[0145] S3031. Determine target features from the features included in the feature information.

[0146] The feature information includes the feature value corresponding to the feature.

[0147] The present invention does not specifically limit the method for determining the target feature from the features included in the feature information. Exemplarily, this step may include: determining the correlation between the features based on the feature values ​​corresponding to the features, and then determining the features whose correlations meet the preset similarity conditions as the target features.

[0148] In practical applications, the embodiment of the present application does not specifically limit the method for obtaining the correlation between the features. For example, the Pearson coefficient method can be used to determine the correlation between any two features.

[0149] Specifically, the correlation between two features can be obtained according to the following formula (1):

[0150]

[0151] Where X and Y are the eigenvalues ​​of feature X and feature Y respectively, cov(X,Y) is the covariance of feature X and feature Y, σ x is the variance of feature X, σ y is the variance of feature Y, μ x is the expectation of feature X, μ y is the expected value of feature Y.

[0152] In practical applications, for the training set of the model, the weaker the correlation between features, the better, that is, ρ X，Y The smaller the value of is, the better, in order to minimize the impact of similarity features on the model results. Therefore, it is necessary to determine the features whose correlation meets the preset similarity conditions as target features.

[0153] In practical applications, on the one hand, the rules that the target features satisfy can be preset, and then the rules that the two features to be determined satisfy can be determined based on the correspondence between the similarity between the two features to be determined and the above rules, thereby determining whether the two features to be determined are target features.

[0154] Specifically, the correlation between features can include the following rules:

[0155] (1)|ρ X，Y |>0.95 There is a significant correlation between features;

[0156] (2)|ρ X，Y |≥0.8 Features are highly correlated;

[0157] (3)0.5≤|ρ X，Y |<0.8 Moderate correlation between features;

[0158] (4)0.3≤|ρ X，Y |<0.5 Low correlation between features;

[0159] (5)|ρ X，Y |<0.3 The relationship between the features is very weak, and the two features are unrelated.

[0160] Exemplarily, the threshold of the preset similarity condition can be set as follows: a feature with at least a low correlation between the features is a target feature. Specifically, after determining the correlation between the features, the target rule is determined based on the correspondence between the correlation and the rule as described above. When the relationship between the two features is low correlation or the relationship between the features is extremely weak, the feature is determined to be the target feature. For example, if the correlation between the two features is 0.2, then according to rule (5), it can be known that the relationship between the two features is extremely weak. In this case, the two features are determined to be target features.

[0161] On the other hand, the preset similarity condition can be set as: the feature with a correlation less than a preset value is the target feature, wherein the embodiment of this application does not specifically limit the size of the preset value. For example, the preset value can be set to 0.5, that is, two features with a correlation less than 0.5 are target features.

[0162] As an optional solution, after obtaining the target features, it is also necessary to perform dimensionality reduction on the obtained targets.

[0163] It should be noted that the embodiments of the present application do not specifically limit the method for performing dimensionality reduction processing on the target features. For example, principal components analysis (PCA) can be used to perform dimensionality reduction processing on the target features. PCA mainly finds the eigenvector corresponding to the largest eigenvalue of the covariance matrix of the data set, thereby finding several directions with the largest data variance, thereby achieving the effect of dimensionality reduction on the data, and replacing the original n target features with a smaller number of m features. The new features are linear combinations of the old features. These linear combinations maximize the sample variance and try to make the new features uncorrelated with each other, ultimately improving the accuracy of the parameter determination model.

[0164] Among them, the specific solution of using PCA technology to reduce the dimension of target features can refer to the solution in the existing technology, which will not be repeated here.

[0165] S3032. Based on the target features and the eigenvalues ​​corresponding to the target features, iteratively train the initial parameters to determine the model.

[0166] In this step, we first need to obtain the characteristic value corresponding to the target feature. It should be noted that the method for obtaining the characteristic value of the target feature is the same as Figure 2 The solutions provided in the embodiment shown are similar, and can be found in detail in Figure 2 The embodiments shown are not described in detail here.

[0167] S304: Determine that the model corresponding to the evaluation index value that meets the preset evaluation conditions is a parameter determination model.

[0168] In some embodiments, after obtaining the parameter determination model through iterative training, the method may further include: determining the parameters of the parameter determination model by adopting a cross-validation method.

[0169] For example, considering that the structures of the initial parameter determination model and the parameter determination model are the same, the verification process is described in detail below by taking the initial parameter determination model and the parameter determination model as the XGBOOST model as an example:

[0170] For the XGBOOST model, the model parameters may include at least one of the following: 'booster', 'objective', 'num_class', 'gamma', 'max_depth', 'subsample', 'min_child_weight', and 'eta'.

[0171] Among them, 'booster': 'gbtree', this parameter has two values, namely tree model and linear model. In the embodiment of this application, the tree model is used as an example;

[0172] 'objective': 'multi:softprob', used to indicate the probability that each feature sample belongs to each category;

[0173] 'num_class': used to indicate the number of categories. For example, the memory of this application ranges from 4G to 48G, and is divided into 2G intervals, including a total of 22 categories;

[0174] 'gamma': used to indicate the degree of fit of the model. The larger the value of the 'gamma' parameter, the more conservative the algorithm is, indicating that the model is less likely to overfit;

[0175] 'max_depth': used to indicate the degree of model fitting. The larger the value of the 'max_depth' parameter, the more likely the model is to overfit;

[0176] 'subsample': used to control the proportion of random sampling of the model. The smaller the 'subsample' value, the more conservative the model is and the less likely it is to overfit.

[0177] 'min_child_weight': used to indicate the degree of model fit. The larger the value of 'min_child_weight', the less likely the model is to overfit.

[0178] 'eta': used to represent the learning rate of the model. The larger the value of 'eta', the less likely the model is to overfit. However, if the value of 'eta' is too large, it will also cause underfitting.

[0179] Assume that the initial parameter determination model contains N parameters, where N is a positive integer. During the validation process, N-1 of the N parameters are fixed. The remaining parameter to be determined is then cross-validated multiple times with different values. The value of the parameter to be determined is determined based on the changes in the evaluation indicators.

[0180] Specifically, the sample data is divided into a training data set and a validation data set according to a preset ratio. It should be noted that the preset ratio is not specifically limited in the embodiment of the present application. For example, the ratio of the training data set to the validation data set can be 7:3. The training data set is used to train the initial parameter determination model, and the validation data set is used to verify the training results during the training process to determine the values ​​of the parameters in the model.

[0181] Furthermore, the model is iteratively trained for the initial parameter determination, wherein the number of iterations is the same as the number of parameter categories, i.e., one round of iterative training is performed for each type of parameter until the parameter of that type is successfully verified, and then iterative training for the next type of parameter is performed. It should be noted that the embodiment of the present application does not specifically limit the training order of multiple types of parameters. For example, the parameters for each training can be randomly determined, or the training can be performed in a preset order.

[0182] For example, taking the case where the parameters include 'booster', 'objective', 'num_class', 'gamma', 'max_depth', 'subsample', 'min_child_weight', and 'eta', there are a total of 8 parameters, that is, the value of N is 8.

[0183] In the first round of iteration, the parameters for training in the first iteration process are first determined. For example, 'gamma' is taken as an example. Then, initial values ​​are set for the seven parameters except the 'gamma' parameter. The initial parameter determination model is trained using the training data set, and the model trained with the training data set is cross-validated using the validation data set. When the accuracy of the parameter determination model reaches the accuracy threshold, the validation passes, and the current value of the 'gamma' parameter is determined to be the value of the 'gamma' parameter in the parameter determination model.

[0184] Furthermore, after the first round of iteration is completed, a second round of iteration is performed. Taking the parameter trained in the second round of iteration as the 'objective' parameter as an example, the other 7 parameters except the 'objective' parameter are fixed, and the 'objective' parameter is trained using the cross-validation method to obtain the value of the 'objective' parameter.

[0185] In this step, on the one hand, for the trained parameters, the values ​​of the parameters can be fixed to the values ​​obtained after training, and the values ​​of the untrained model parameters can be fixed to the initial values. For example, in combination with the above, in the first round of iterative training, the value of the 'gamma' parameter is 0.1, then in this round of training, the value of the 'gamma' parameter is fixed to 0.1.

[0186] On the other hand, you can also set all 7 parameters except the 'objective' parameter to initial values. For example, the value of the 'gamma' parameter and the values ​​of other parameters can be fixed to the initial values.

[0187] Furthermore, the remaining parameters are iteratively trained according to the above training method until the values ​​of all parameters in the parameter determination model are determined, and the parameter determination model can be determined according to the values ​​of the parameters.

[0188] It should be noted that the embodiments of the present application do not specifically limit the cross-validation method. Exemplarily, the cross-validation method may include at least one of the following: simple cross-validation, 2-fold cross-validation (2-CV), K-fold cross-validation (K-CV), leave one out (LOO), hand-out cross validation, etc. It is understandable that the process of performing validation using the above cross-validation method can refer to the existing technology and will not be repeated here.

[0189] In this solution, the values ​​of the parameters to be determined are determined through cross-validation, thereby obtaining a parameter determination model. This can make full use of limited data to find appropriate parameters, prevent the parameter determination model from overfitting, and improve the accuracy of the parameter determination model.

[0190] S305: Obtain target features corresponding to the target Spark task.

[0191] S306: Parameterize the target feature to obtain the encoding value of the target feature.

[0192] S307: Input the encoded value into the parameter determination model to obtain the parameters of the target Spark task.

[0193] S308. Set the parameters of the target Spark task; or adjust the current parameters of the target Spark task to parameters obtained through the parameter determination model.

[0194] It should be noted that steps S305 to S308 are Figure 2 The solutions in the illustrated embodiment are similar, and the specific solutions can be referred to Figure 2 The embodiments shown are not described in detail here.

[0195] The data processing method provided in the embodiment of the present application obtains the historical operation information of the Spark task, extracts the characteristic information of the Spark task from the historical operation information, iteratively trains the initial parameter determination model based on the characteristic information of the Spark task, until the calculated evaluation index value meets the preset evaluation condition, obtains the parameter determination model, obtains the target feature corresponding to the target Spark task, parameterizes the target feature, obtains the encoding value of the target feature, inputs the encoding value into the parameter determination model, obtains the parameters of the target Spark task, and sets the parameters of the target Spark task; or, adjusts the current parameters of the target Spark task to the parameters obtained by the parameter determination model. Through this solution, the Spark task parameters can be automatically adjusted according to the parameter determination model, which can improve the efficiency of obtaining the Spark task parameters while reducing labor costs. In addition, in the process of training the parameter determination model, the model parameters of the parameter determination model are obtained by cross-validation, which can improve the accuracy of the model parameters and thus improve the accuracy of the parameter determination model.

[0196] Figure 4 A flow chart of a method for training a parameter determination model provided in one embodiment of the present application. The present application embodiment provides a method for training a parameter determination model, which is applied to a training device for a parameter determination model, and the device can be implemented in software and / or hardware. Optionally, the training device for the parameter determination model can be integrated into a server, for example, the training device for the parameter determination model is a chip or circuit in the server; or, the training of the parameter determination model is a server. Next, an example is given using the server as the execution subject. Among them, the server of the embodiment of the present application can be Figure 1 The server shown in the figure may also be other servers, and this embodiment of the application does not specifically limit this.

[0197] like Figure 4 As shown, the information processing method provided in the embodiment of the present application may include the following steps:

[0198] S401: Obtain historical running information of Spark tasks.

[0199] S402: Extract characteristic information of the Spark task from historical operation information.

[0200] S403: Iteratively train the initial parameter determination model based on the characteristic information of the Spark task until the calculated evaluation index value meets the preset evaluation condition.

[0201] The initial parameter determination model has the same structure as the parameter determination model, and the parameter determination model is used to predict the parameters of the Spark task.

[0202] The specific solution of S403 is described in detail below in conjunction with steps S4031 to S4032:

[0203] S4031. Determine target features from features included in the feature information;

[0204] Determine the correlation between the features based on the feature values ​​corresponding to the features;

[0205] The features whose relevance meets the preset similarity conditions are determined as target features.

[0206] S4032. Based on the target features and the feature values ​​corresponding to the target features, iteratively train the initial parameter determination model to obtain a parameter determination model.

[0207] In each round of iteration, N-1 model parameters out of N model parameters are fixed, and multiple cross-validations are performed on the remaining model parameter to be determined using different values. The value of the model parameter to be determined is determined based on the changes in the evaluation index.

[0208] Furthermore, a parameter determination model is obtained according to the values ​​of the model parameters to be determined.

[0209] S404: Determine that the model corresponding to the evaluation index value that meets the preset evaluation conditions is a parameter determination model.

[0210] It should be noted that Figure 4 The scheme and beneficial effects of steps S401 to S404 in the embodiment shown are similar to those in FIG. Figure 3 Steps S301 to S304 in the embodiment shown are similar, and can be found in detail. Figure 3 The embodiments shown are not described in detail here.

[0211] Figure 5 This is a structural diagram of an information processing device provided by an embodiment of the present application. This embodiment of the present application provides an information processing device, which can be integrated into an electronic device such as a server. Figure 5 As shown, the information processing device 500 includes: an acquisition module 501 and a determination module 502.

[0212] Acquisition module 501, acquires target features corresponding to the target Spark task;

[0213] The determination module 502 is used to determine the parameters of the target Spark task according to the target characteristics and parameter determination model. The parameter determination model is used to predict the parameters of the Spark task. The parameter determination model is trained based on the historical operation information of the Spark task.

[0214] The device provided in the embodiment of the present application can be used to perform Figure 2 The implementation principles and technical effects of the methods in the illustrated embodiments are similar and will not be described in detail here.

[0215] In a possible implementation, the determining module 502 may be specifically configured to:

[0216] Parameterize the target feature and obtain the encoding value of the target feature;

[0217] Input the encoded value into the parameter determination model to obtain the parameters of the target Spark task.

[0218] In a possible implementation, the determining module 502 is further configured to:

[0219] Set the parameters of the target Spark task; or,

[0220] Adjust the current parameters of the target Spark task to the parameters obtained through the parameter determination model.

[0221] In one possible implementation, the parameter determination model is trained by the following steps:

[0222] Get the historical running information of Spark tasks;

[0223] Extract Spark task feature information from historical running information;

[0224] Iteratively train the initial parameter determination model based on the characteristic information of the Spark task until the calculated evaluation index value meets the preset evaluation conditions;

[0225] Determine that the model corresponding to the evaluation index value when it meets the preset evaluation conditions is the parameter determination model;

[0226] The structure of the initial parameter determination model is the same as that of the parameter determination model.

[0227] It should be noted that the device provided in the embodiment of the present application can be used to perform Figures 2 and 3 The implementation principles and technical effects of the methods in the illustrated embodiments are similar and will not be described in detail here.

[0228] Figure 6This is a structural diagram of a parameter determination model training device provided by an embodiment of the present application. This embodiment of the present application provides a parameter determination model training device, which can be integrated into an electronic device such as a server. Figure 6 As shown, the training device 600 for parameter determination model includes: an acquisition module 601, an extraction module 602, a training module 603 and a determination module 604.

[0229] The acquisition module 601 is used to obtain the historical running information of the Spark task;

[0230] Extraction module 602, used to extract characteristic information of Spark tasks from historical operation information;

[0231] A training module 603 is configured to iteratively train an initial parameter determination model based on the characteristic information of the Spark task until the calculated evaluation index value meets a preset evaluation condition;

[0232] A determination module 604 is configured to determine that the corresponding model when the evaluation index value satisfies a preset evaluation condition is a parameter determination model;

[0233] The initial parameter determination model has the same structure as the parameter determination model, and the parameter determination model is used to predict the parameters of the Spark task.

[0234] The device provided in the embodiment of the present application can be used to perform Figure 4 The implementation principles and technical effects of the methods in the illustrated embodiments are similar and will not be described in detail here.

[0235] In one possible implementation, the feature information includes feature values ​​corresponding to the features, and the training module 603 is specifically configured to:

[0236] determining target features from features included in the feature information;

[0237] Based on the target features and the eigenvalues ​​corresponding to the target features, the initial parameters are iteratively trained to determine the model.

[0238] In one possible implementation, when the training module 603 is used to determine the target feature from the features included in the feature information, it is specifically used to:

[0239] Determine the correlation between the features based on the feature values ​​corresponding to the features;

[0240] The features whose relevance meets the preset similarity conditions are determined as target features.

[0241] In a possible implementation, the parameter determination model training device 600 further includes a verification module (not shown) configured to determine the parameters of the parameter determination model by cross-validation after the parameter determination model is obtained through iterative training.

[0242] The device provided in the embodiment of the present application can be used to perform Figure 4 The implementation principles and technical effects of the methods in the illustrated embodiments are similar and will not be described in detail here.

[0243] It should be noted that it should be understood that the division of the various modules of the above device is merely a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. Moreover, these modules can all be implemented in the form of software called by a processing element; or they can all be implemented in the form of hardware; or some modules can be implemented in the form of software called by a processing element, and some modules can be implemented in the form of hardware. For example, the processing module can be a separately established processing element, or it can be integrated into a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called by a processing element of the above device to perform the functions of the above processing module. The implementation of other modules is similar. In addition, these modules can all or partly be integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed by the hardware integrated logic circuit in the processor element or by instructions in the form of software.

[0244] For example, the above modules can be one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (ASICs), one or more microprocessors (DSPs), or one or more field programmable gate arrays (FPGAs). For another example, when a module is implemented by scheduling program code through a processing element, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules can be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0245] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)).

[0246] Figure 7 This is a schematic diagram of the structure of an electronic device provided in one embodiment of the present application. Figure 7 As shown, the electronic device 700 may include: a processor 701, a memory 702, a communication interface 703, and a system bus 704. The memory 702 and the communication interface 703 are connected to the processor 701 via the system bus 704 and communicate with each other. The memory 702 is used to store instructions, the communication interface 703 is used to communicate with other devices, and the processor 701 is used to call instructions in the memory to execute the solution of the embodiment of the dangerous goods identification method described above.

[0247] Should Figure 7 The system bus 704 mentioned in the figure can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The system bus 704 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0248] The communication interface 703 is used to implement communication between the database access apparatus and other devices (such as a client, a read-write library, and a read-only library).

[0249] The memory 702 may include a random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0250] The processor 701 can be a general-purpose processor, including a central processing unit, a network processor (NP), etc.; it can also be a digital signal processor DSP, an application-specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0251] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program runs on an electronic device, the electronic device executes the information processing method and / or parameter determination model training method of any of the above method embodiments.

[0252] An embodiment of the present application also provides a chip for executing instructions, which is used to execute the information processing method and / or parameter determination model training method of any of the above method embodiments.

[0253] An embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When the at least one processor executes the computer program, it can implement the information processing method and / or parameter determination model training method of any of the above method embodiments.

[0254] In this application, "at least one" means one or more, and "more" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship; in the formula, the character " / " indicates that the previous and next associated objects are in a "division" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0255] It is understood that the various numerical numbers involved in the embodiments of the present application are only for the convenience of description and are not intended to limit the scope of the embodiments of the present application. In the embodiments of the present application, the order of the sequence numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0256] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An information processing method, characterized in that: include: Get the target features corresponding to the target Spark task; Using a one-hot encoding method, parameterize the target feature to obtain an encoding value of the target feature; Inputting the encoded value into a parameter determination model to obtain parameters of the target Spark task, wherein the parameter determination model is used to predict the parameters of the Spark task, and the parameter determination model is trained based on historical operation information of the Spark task; If the target Spark task is used for the first time, setting the obtained parameters of the target Spark task as the parameters of the target Spark task; If the target Spark task is not used for the first time, adjusting the current parameters of the target Spark task to parameters obtained by the parameter determination model; The parameter determination model is obtained by training through the following steps: Get the historical running information of the Spark task, which is obtained through the Rest API; Extracting feature information of the Spark task from the historical running information, where the feature information includes feature values ​​corresponding to the features; Determining the correlation between the features based on the feature values ​​corresponding to the features included in the feature information of the Spark task; The features whose correlations meet the preset similarity conditions are determined as target features, and the principal component analysis technology PCA is used to reduce the dimension of the target features; Based on the target features of the dimensionality reduction process and the eigenvalues ​​corresponding to the target features of the dimensionality reduction process, the initial parameter determination model is iteratively trained until the calculated evaluation index value meets the preset evaluation conditions; The model corresponding to the evaluation index value when it satisfies the preset evaluation condition is determined to be the parameter determination model, and the parameters of the parameter determination model are determined by cross-validation.

2. The method according to claim 1, characterized in that in, The initial parameter determination model has the same structure as the parameter determination model.

3. An information processing device, characterized in that include: The acquisition module is used to obtain the encoding value of the target feature corresponding to the target Spark task; A determination module is used to parameterize the target feature using a one-hot encoding method to obtain an encoding value of the target feature; Inputting the encoded value into a parameter determination model to obtain parameters of the target Spark task, wherein the parameter determination model is used to predict the parameters of the Spark task, and the parameter determination model is trained based on historical operation information of the Spark task; The determining module is further configured to: If the target Spark task is used for the first time, setting the obtained parameters of the target Spark task as the parameters of the target Spark task; If the target Spark task is not used for the first time, adjusting the current parameters of the target Spark task to parameters obtained by the parameter determination model; The parameter determination model is obtained by training through the following steps: Get the historical running information of the Spark task, which is obtained through the Rest API; Extracting feature information of the Spark task from the historical running information, where the feature information includes feature values ​​corresponding to the features; Determining the correlation between the features based on the feature values ​​corresponding to the features included in the feature information of the Spark task; The features whose correlations meet the preset similarity conditions are determined as target features, and the principal component analysis technology PCA is used to reduce the dimension of the target features; Based on the target features of the dimensionality reduction process and the eigenvalues ​​corresponding to the target features of the dimensionality reduction process, the initial parameter determination model is iteratively trained until the calculated evaluation index value meets the preset evaluation conditions; The model corresponding to the evaluation index value when it satisfies the preset evaluation condition is determined to be the parameter determination model, and the parameters of the parameter determination model are determined by cross-validation.

4. An electronic device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to claim 1 or 2 is implemented.

5. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is run on an electronic device, the electronic device executes the method according to claim 1 or 2.

6. A computer program product comprising a computer program, characterized in that When the computer program is run on an electronic device, the electronic device is caused to perform the method according to claim 1 or 2.

Images