Production resource prediction method and device for data element

Abstract

The invention relates to the technical field of data processing, in particular to a production resource prediction method for a data element, which comprises the following steps of: in response to a received instruction issued by a new element production task, performing preliminary analysis on the new element production task to determine that the new element production task can be executed; reading an element resource prediction model corresponding to the new element production task from an element resource prediction model library; generating an initial prediction result through the element resource prediction model by taking task data carried in the new element generation task as input; and performing intelligent model recommendation on the initial prediction result through intelligent optimization algorithm recommendation, and generating production resources of the element production task corresponding to the new element. By applying the method, the element production resource prediction period can be greatly shortened by using very low cost, and the efficiency and the result of the production environment are not influenced.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method and device for predicting production resources of data elements. Background Art

[0002] In today's data age, we are generating a large amount of data every day, which brings huge opportunities and challenges. On the one hand, the large-scale collection and analysis of data enables us to better understand and predict human behavior, social trends and market demand, which provides more decision-making basis and innovation opportunities for enterprises, governments and individuals. On the other hand, the security and compliance of data use has gradually become an important issue. Large-scale data collection and abuse have put personal privacy data at greater risk.

[0003] Therefore, data protection and processing technologies are being used more and more widely in various data sources. As one of the data protection and processing technologies, data components can not only ensure that data does not leave the domain and avoid data leakage risks, but also convert raw data into valuable data component results, promote application innovation, improve work efficiency, and improve service quality. The production of components requires computing resources (CPU, memory). When faced with a high-concurrency scenario with a large number of component production tasks, the setting of computing resource quotas for each component production task will become an important factor affecting component production efficiency and task operation performance.

[0004] At present, there are several ideas for resource quota methods for data protection processing technology similar to component production: (a) manual configuration based on experience. When a new task is cold started, the developer will allocate resources to the task based on experience, using a fixed value. Each subsequent scheduled restart of the task will allocate resources according to the initial fixed value. If problems such as task timeout / failure occur, the developer will modify the task resource quota (generally increase resources) and reissue the component production task to ensure that the task can be completed in a timely manner. (b) Recommend configuration based on statistical methods of historical records. By statistically analyzing the load data of historical task executions, the historical load peak is taken as the resource quota for the next task execution; (c) Intelligently recommend configuration based on machine learning methods. Use task execution records and performance indicators and other parameters as training sets to train the machine learning model, and then intelligently recommend resource quotas based on the machine learning model.

[0005] However, manual configuration based on experience solves the discrepancy between resource application and actual resource usage by overselling resources, which results in low resource utilization of the task itself, and may not meet the demand for re-running the task as the amount of data grows. The recommended values ​​given by statistical methods for configuration are generally too high, which increases the cost of task operation. Recommendations based on machine learning can better meet the needs, but require a large amount of task operation history records to train the model, which makes the time required for the recommendation to take effect very long, and loses the value of the function.

[0006] In view of this, how to provide a production resource prediction method for data components and greatly shorten the model training time has become a technical problem that needs to be solved urgently. Summary of the invention

[0007] The embodiments of the present application provide a data element production resource prediction method, a data element production resource prediction device, an electronic device and a computer storage medium, which are used to solve the problem of how to configure production resources in the data element production process and thus improve resource utilization.

[0008] In a first aspect of an embodiment of the present application, a method for predicting production resources of a data element is provided, comprising:

[0009] In response to receiving an instruction issued by a new component production task, after preliminary analysis of the new component production task and confirming that the new component production task is executable, a component resource prediction model corresponding to the new component production task is read from a component resource prediction model library, wherein the component resource prediction model library is a collection of component resource prediction models obtained by training a simulation result obtained by simulating the component on-line by a notification simulation system through a transfer learning training model when it is detected that the component production task is on-line, and updating the prediction model in combination with an online model generated by training historical instance data through an online learning model;

[0010] Taking the task data carried in the new component generation task as input, generating an initial prediction result through the component resource prediction model;

[0011] The initial prediction results are intelligently recommended by model through optimization algorithm, and production resources of the component production task corresponding to the production resources of the new component are generated, wherein the model intelligent recommendation is to perform performance prediction through training and generalization of machine learning models, or to determine the final recommendation value through the recommendation parameter optimization method.

[0012] In a second aspect of an embodiment of the present application, a production resource prediction device for a data element is provided, comprising:

[0013] A reading module is configured to respond to receiving an instruction issued by a new component production task, and after preliminary analysis of the new component production task and confirming that the new component production task is executable, read the component resource prediction model corresponding to the new component production task from the component resource prediction model library, wherein the component resource prediction model library is a collection of component resource prediction models obtained by updating the prediction model by combining the online model generated by training the historical instance data with the transfer learning training model when detecting that the component production task is online;

[0014] A first generation module is configured to take the task data carried in the new component generation task as input and generate an initial prediction result through the component resource prediction model;

[0015] The second generation module is configured to perform model intelligent recommendation on the initial prediction result through intelligent recommendation of optimization algorithm, and generate production resources of the component production task corresponding to the production resources of the new component, wherein the model intelligent recommendation is to perform performance prediction through training and generalization of machine learning models, or to determine the final recommendation value through the recommendation parameter optimization method.

[0016] In a third aspect of an embodiment of the present application, a computing device is provided, including:

[0017] Memory and processor;

[0018] The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the production resource prediction method of the data element are implemented.

[0019] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the production resource prediction method for the above-mentioned data element are implemented.

[0020] The present application provides a method for predicting production resources of data components, comprising: first, in response to receiving an instruction issued by a new component production task, after preliminary analysis of the new component production task, confirming that the new component production task is executable, reading a component resource prediction model corresponding to the new component production task from a component resource prediction model library, wherein the component resource prediction model library is a simulation model generated by training a simulation result obtained by simulating the component on the line by a notification simulation system through a transfer learning training model when it is detected that the component production task is online, and updating the prediction model in combination with an online model generated by training historical instance data through an online learning model, thereby obtaining a set of component resource prediction models; then, taking the task data carried in the new component generation task as input, an initial prediction result is generated through the component resource prediction model; finally, the initial prediction result is model intelligently recommended through the optimization algorithm intelligent recommendation, and the production resources of the component production task corresponding to the production resources of the new component are generated, wherein the model intelligent recommendation is to perform performance prediction through a trained and generalized machine learning model, or to determine the final recommendation value through a recommended parameter optimization method.

[0021] By using the production resource prediction method of the data components provided in the embodiment of the present application, the component production resource prediction cycle can be greatly shortened at a very low cost without affecting the efficiency and results of the production environment. The present application avoids the long recording cycle of a large amount of historical data required for model training through a simulation system, and adopts data sampling during simulation, which reduces the running cost of simulation tasks and improves the concurrency of simulation tasks. The present application fits the resource distribution of component production tasks through machine learning, and achieves recommendation results that are better than manual experience and statistical methods. The present application generalizes the characteristics of component production in different environments through transfer learning, thereby improving the applicability of recommendation results in different environments.

[0022] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, which can be implemented in accordance with the contents of the specification, and in order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0024] Figure 1 A schematic diagram of a framework of a production resource prediction system for a data element provided in an embodiment of the present application;

[0025] Figure 2 A schematic diagram of a flow chart of a method for predicting production resources of a data element provided in an embodiment of the present application;

[0026] Figure 3 A flowchart of a simulation calculation execution method in a production resource prediction method for a data element provided in an embodiment of the present application;

[0027] Figure 4 A timing diagram of a simulation calculation execution method in a production resource prediction method for a data element provided in an embodiment of the present application;

[0028] Figure 5 A flow chart of a model training and generalization method based on transfer learning in a production resource prediction method for a data element provided in an embodiment of the present application;

[0029] Figure 6 A schematic diagram of the structure of a production resource prediction device for a data element provided in an embodiment of the present application;

[0030] Figure 7 A structural block diagram of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0032] See also Figure 1 , Figure 1 A schematic diagram of the framework of a production resource prediction system for a data element provided in an embodiment of the present application.

[0033] like Figure 1 As shown, the production resource prediction system of the data element includes three parts: simulation service, algorithm training service and algorithm prediction service.

[0034] Specifically, the simulation service part is used to execute the simulation task after receiving the simulation task. After receiving the simulation instance and configuration parameters sent by the simulation controller, the simulation system first preprocesses the received task data based on the task executor, including parameter processing, sampling data processing, component processing and algorithm data processing; then performs sampling service, that is, configures the corresponding environment preparation for the task data obtained after preprocessing, and samples the task data; secondly, performs data service, that is, generates statistical table information based on the sampled data, wherein the statistical table information carries source table information and sampling table information; finally, performs data persistence processing on the data that completes the simulation service and stores it in the database.

[0035] The algorithm training part is used to train the model and generate a component resource prediction model after monitoring the task online or task update, wherein the component resource prediction model is obtained based on the update of the online model and the simulation model. Specifically, after monitoring the instruction of the task online or task update, first, the simulation instance and configuration parameters are sent to the simulation system through the simulation control controller to trigger the simulation system to execute the simulation task. When the number of executions of the simulation task reaches the preset number of executions, the simulation result is trained by the transfer learning model to generate a simulation model, and the initial prediction model is updated by the simulation model to obtain the first prediction model; then, the historical instance data is obtained, and the historical instance data is trained by the online learning model to generate an online model, and the initial prediction model is updated by the online model to obtain the second prediction model; finally, based on the first prediction model and the second prediction model, a prediction model is generated, the prediction model is cached in the model, and the task data in the algorithm training service process is persisted and stored in the database.

[0036] In addition, in the algorithm training part, when the simulation controller sends simulation instances and configuration parameters to the simulation system, it also performs monitoring services, which include monitoring the actual load and actual status of the task; on the other hand, it also monitors in real time through the task queue whether there are other new tasks online or updated.

[0037] The algorithm prediction part is used to determine whether the task is executable when the monitoring task is issued. If the task is executable, determine whether there is a model that matches the task. If there is a model that matches the task, the machine learning algorithm is used to predict and generate an initial prediction result. Finally, the initial prediction result is processed through the optimization algorithm intelligent recommendation to generate the production resources for the component production task.

[0038] When the task cannot be executed, a cold start is started to generate production resources for the component production task.

[0039] If there is no model matching the task, the production resources for the component production task are directly generated.

[0040] The production resource prediction system for data components provided by the embodiment of the present application can greatly shorten the component production resource prediction cycle at a very low cost without affecting the efficiency and results of the production environment. The present application avoids the long recording cycle of a large amount of historical data required for model training through a simulation system, and adopts data sampling during simulation, which reduces the running cost of simulation tasks and improves the concurrency of simulation tasks. The present application fits the resource distribution of component production tasks through machine learning, and achieves recommendation results that are better than manual experience and statistical methods. The present application generalizes the characteristics of component production in different environments through transfer learning, thereby improving the applicability of recommendation results in different environments.

[0041] See also Figure 2 , Figure 2 A flowchart of a method for predicting production resources of a data element provided in an embodiment of the present application. Figure 2 As shown, the specific steps include the following steps.

[0042] Step S202: In response to receiving an instruction to issue a new component production task, after preliminary analysis of the new component production task, confirming that the new component production task is executable, read the component resource prediction model corresponding to the new component production task from the component resource prediction model library, wherein the component resource prediction model library is a simulation model generated by training the simulation results obtained by simulating the components that are notified to the simulation system when the component production task is detected to be online, and updating the prediction model in combination with the online model generated by training the historical instance data with the online learning model, thereby obtaining a collection of component resource prediction models.

[0043] Step S204: taking the task data carried in the new component generation task as input, generating an initial prediction result through the component resource prediction model.

[0044] Step S206: Perform model intelligent recommendation on the initial prediction result through intelligent recommendation of optimization algorithm, and generate production resources of the component production task corresponding to the production resources of the new component, wherein the model intelligent recommendation is to perform performance prediction through training and generalization of machine learning models, or to determine the final recommendation value through recommendation parameter optimization method.

[0045] In the embodiment of the present application, before the response to receiving the instruction issued by the new component production task includes:

[0046] In response to monitoring an instruction for launching a new component task or updating a new component task, triggering a simulation controller to preprocess the component corresponding to the instruction for launching a new component task or updating the new component task, and generating configuration data corresponding to the component, wherein the preprocessing includes parameter processing, sample data processing, component processing and algorithm data processing;

[0047] Based on the configuration data and simulation instance corresponding to the component, informing the simulation system to simulate the new component and generate a simulation result, and using the simulation result as input to train the transfer learning training model to generate a simulation model;

[0048] Calling historical instance data, taking the historical instance data as input, training the historical instance data through an online learning model, and generating an online model;

[0049] A component resource prediction model is generated based on a first prediction model obtained by updating the prediction model based on the simulation model, and a second prediction model obtained by updating the prediction model based on the online model, wherein the component resource prediction model caches a model corresponding to the task, and at least two component resource prediction models constitute a component resource prediction model library.

[0050] Specifically, the notifying a simulation system to simulate the new component based on the configuration data and the simulation instance corresponding to the component includes:

[0051] Assembling task information corresponding to the production task in a specific data format through a message queue, wherein the task information includes configuration data and a simulation instance;

[0052] Based on the assembled task information, monitor in real time whether there is a new task online in the message queue. When a new task is detected online in the message queue, parse the task parameters of the new task and store the task data persistently, wherein the task parameters carry the actual task load and task execution status;

[0053] Identify the affinity characteristics of the data element corresponding to the production task to the production resource, and based on the identification result, use a binary classification algorithm to determine the resource characteristics corresponding to the data element, wherein the resource characteristics include a filter element and a non-filter element, and the filter element corresponds to the data element as the minimum resource element, and filter the data element. The binary classification algorithm is trained by a training set composed of the data volume characteristics and the computational complexity characteristics of the element and the label result characteristics of whether filtering is required;

[0054] When it is determined that the data element is a non-filtering element, the data element is added to a task execution queue of a simulation queue for simulation until the number of simulations reaches a preset simulation number threshold.

[0055] It should be noted that in the embodiment of the present application, the affinity of component resources and whether to filter are identified, and the classification algorithm adopts a binary classification algorithm, deep learning, GBDT-type algorithm, etc. The data features include component type features, data set features, code features, and resource consumption features, and the components are divided into components that use minimum resources and components that do not use minimum resources.

[0056] More specifically, when determining that the data element is a non-filtering element, adding the data element to a task execution queue of a simulation queue for simulation includes:

[0057] Sorting according to a specific strategy, determining the task execution order of the simulation components to be executed in the task execution queue, wherein the specific strategy is to determine the task simulation priority and the priority of the newly added task according to the component execution cycle, and processing through the target optimization algorithm, inserting the generated optimization order into the queue, and dynamically adjusting the optimization order strategy in combination with resource constraints;

[0058] Based on the task execution sequence, the simulation element to be executed is simulated, and the execution process indicators of the simulation element are monitored and the corresponding indicator data are recorded, wherein the process indicators include data characteristics, environment characteristics, configuration characteristics and execution characteristics.

[0059] Simulation resources are dynamically adjusted based on resource limitations through algorithms, which can improve overall resource utilization.

[0060] It should be noted that the data features here include: (1) Data features: including the number of rows in the data set, the number of columns in the data set, the number of bytes, and the number of samples; (2) Environmental features: including the number of computing nodes, the total number of computing cores, CPU frequency, total memory size, memory running speed, the number of available computing cores at the time of task submission, and the available memory size at the time of task submission; (3) Configuration features: including the number of cores, the number of memories, and the number of task workers; (4) Execution features: including execution time, execution status, the actual number of cores consumed, and the actual memory consumed.

[0061] It should be noted that the method for calculating the resource limitation includes: determining the number of computing nodes and the resource quota corresponding to each of the computing nodes, and obtaining the total resources based on the resource quota corresponding to each of the computing nodes, wherein the total resources carry simulation task resources waiting to be executed, simulation task resources being executed, and simulation task resources that have been completed, and the simulation task resources being executed include simulation task resources that have been submitted but not executed and simulation task resources being executed.

[0062] In actual applications, the total computing resources in the total resource limit of the simulation system are the static resources of the computing nodes. It is necessary to know the number of computing nodes and the resource quota of each computing node, and sum them up to get the total resources; the simulation task resources waiting to be executed are the sum of the resource quotas of the tasks that have not yet started to be executed in the simulation queue; except for the tasks waiting to be executed and the tasks that have been completed, all are tasks that are being executed, including tasks that have been submitted but not executed and tasks that are being executed. If the production task load is monitored to be high, it is necessary to compress the simulation computing resources to ensure the efficient execution of production tasks.

[0063] More specifically, when it is determined that the data element is a non-filtering element, adding the data element to a task execution queue of a simulation queue for simulation includes:

[0064] Using a target data sampling algorithm to perform data sampling on the data element, and writing the sampling results into a sampling statistics table, wherein the target data sampling algorithm is a sampling algorithm that retains the distribution of the source data;

[0065] Based on the sampling statistics table, source table information and sampling table information are obtained.

[0066] It should be noted that the sampling algorithms here include but are not limited to stratified sampling, SMOT sampling, etc.

[0067] Specifically, the simulation result is used as input, and a transfer learning training model is used for training to generate a simulation model, including:

[0068] Accessing and reading the simulation results generated after the new component production simulation execution process is completed, wherein the simulation results carry a simulation data set, and the simulation data set carries component data set characteristics, environment characteristics, configuration characteristics, and execution characteristics;

[0069] Modeling training and learning are performed on component historical data, an initial simulation model is constructed based on specific hyperparameters, a training set, a test set, and a validation set are divided, the initial simulation model is trained based on the training set and evaluation parameters are calculated, and the model is persisted after the initial simulation model training converges;

[0070] Collect statistics on the task execution data of the component in the simulation environment, and trigger the transfer learning update process when the number of simulations reaches a preset simulation number threshold;

[0071] Access and read the trained initial simulation model, access and read the real production data set, adaptively update the initial simulation model to the production environment model based on a specific transfer learning algorithm, update the production environment model and calculate the evaluation parameters, and persist the model after the production environment model update converges, wherein the transfer learning algorithm includes a long short-term memory network model pre-trained by simulation, and an algorithm framework formed by fine-tuning and adaptive learning model update based on a small number of samples in the production environment; after running multiple times, the production environment model is automatically updated based on the target update strategy.

[0072] It should be noted that the initial simulation model is constructed based on specific hyperparameters, and the training model is divided into training set, test set and validation set, which supports multiple machine learning models. For example, the deep learning model multi-layer perceptron (MLP) is used to fit the mapping between input features and output features. This process can be described by the formula:

[0073]

[0074] Where L represents the number of layers of MLP, f l (·) represents the mapping function of the lth hidden layer, represents the model prediction result, d, s, c, e represent data features, environment features, configuration features, and execution features respectively. The activation function of each layer mapping function is designed as ReLU: f l (x) = ReLU(W MLP,l x+b MLP,l ), where W MLP,l represents the learning weight matrix of the input of layer l, b MLP,l Represents the bias vector of the lth layer.

[0075] More specifically, the model persistence method includes:

[0076] Calculate the model’s adaptation loss based on the difference between the model’s predictions and the true statistics;

[0077] By constructing an optimizer, optimizing the adaptation loss, and updating the model parameters.

[0078] It should be noted that the core idea of ​​the transfer learning model here is to calculate the model's adaptation loss based on the difference between the model's prediction results and the actual statistical data, and then update the model parameters by constructing an optimizer to optimize the adaptation loss.

[0079] In the embodiment of the present application, the intelligent recommendation of the optimization algorithm is used to perform model intelligent recommendation on the initial prediction result to generate production resources of the component production task corresponding to the production resources of the new component, including:

[0080] Constructing a solution space by an optimization algorithm, wherein the solution space carries multiple sets of component production task input features generated by the algorithm;

[0081] Determine, by a machine learning model, a prediction result corresponding to each set of input features in the solution space, and determine, based on the prediction result and a production task execution cost calculation method, a component production task execution cost corresponding to the production resource of the new component;

[0082] The solution space is searched by an optimization algorithm to determine the production resource prediction result corresponding to the minimum production task execution cost among the component production task execution costs, and based on the production resource prediction result, the production resources corresponding to the component production task of the new component are generated.

[0083] It should be noted that the method for calculating the component production resource consumption cost provided in the embodiment of the present application is the execution cost of the task, wherein the task execution cost here can be defined as the weighted sum of the product of different types of resource quotas and durations of the Master and Worker. The formula is as follows:

[0084]

[0085] Among them, Ci represents the task execution cost, w represents the resource weight, N represents the number of workers, and y represents the task execution time.

[0086] In the embodiment of the present application, the task resource recommendation includes a genetic algorithm, which is used to perform iterative search on a large scale of feasible solutions through crossover and mutation, using the model prediction value as the result of the fitness function, and finally obtain the optimal solution.

[0087] See also Figure 3 , Figure 3 The flowchart of the simulation calculation execution method in the production resource prediction method of a data element provided in the embodiment of the present application is as follows. Figure 3 As shown, the specific steps include the following steps.

[0088] Step S302: When a new component production task is launched, the simulation system is notified to simulate the component. When notifying, the message queue will assemble the task information in a specific data format. The simulation system will monitor whether there is a task online in the message queue, parse the task parameters, and persist the task data.

[0089] Step S304: the simulation system identifies the resource affinity characteristics of the data element and determines whether to simulate a filtering element. If it is identified as a filtering element, it does not participate in subsequent steps and is determined to be a minimum resource element.

[0090] Step S306: After confirming that the component needs to be simulated in step S304, the simulation system puts the component into the simulation task queue. The simulation queue sorts all simulation components waiting to be executed according to a specific strategy, gives the execution order of all tasks, and executes the component simulation in sequence;

[0091] Step S308: the simulation component executed in step S306 first uses a specific data sampling algorithm to perform data sampling, writes the sampling results into the sampling table, and obtains source table information and sampling table information;

[0092] Step S310: before component simulation, a simulation environment preparation process is performed, including computing environment, execution parameters, and sampling data, etc.;

[0093] Step S312: Execute component production simulation, monitor the simulation component execution process indicators and record the indicator data;

[0094] Step S314: After a component simulation is completed, determine whether to continue the simulation. If it needs to be executed again, repeat steps S306 to S312. If not, the component exits the component production simulation process.

[0095] See also Figure 4 , Figure 4 A timing diagram of a simulation calculation execution method in a production resource prediction method for a data element provided in an embodiment of the present application.

[0096] See also Figure 5 , Figure 5 The present invention provides a flow chart of a method for predicting production resources of a data element based on transfer learning and a generalization method. Figure 5 As shown, the specific steps include the following steps.

[0097] Step S502: After the component production simulation execution process is completed, the simulation data set is accessed and read. The data set contains N samples, including four types of features: component data set features, environment features, configuration features, and execution features. Each type of feature contains multiple feature tags. For example, the environment feature includes the number of computing nodes, the total number of computing cores, the CPU frequency, the total memory size, the memory running speed, the number of available computing cores at the time of task submission, and the available memory size at the time of task submission.

[0098] Step S504: Perform modeling training and learning on the component historical data, build a model based on specific hyperparameters, divide the training set, test set and validation set, train the model and calculate the evaluation parameters, and persist the model after the model training converges;

[0099] Step S506: collecting statistics on the task execution data of the component in the simulation environment, and triggering the transfer learning update process after accumulating a specific number of executions;

[0100] Step S508: access and read the trained simulation model, access and read the real production data set, adaptively update the simulation model to the production environment model based on a specific transfer learning algorithm, update the model and calculate the evaluation parameters, and persist the model after the model update converges;

[0101] Step S510: After the component resource prediction model training and generalization are completed, new component production task resources will be intelligently recommended based on the model. The model recommendation process predicts the performance through the above-mentioned trained and generalized machine learning model, and also determines the final recommendation value through the recommended parameter optimization method, specifically the genetic algorithm. The parameter optimization process searches for the optimal solution by minimizing the component production resource consumption cost;

[0102] Step S512: The component production environment model will be automatically updated based on a specific strategy after running multiple times to ensure that the intelligent model can always adapt to changes in component production tasks.

[0103] By using the production resource prediction method of the data components provided in the embodiment of the present application, the component production resource prediction cycle can be greatly shortened at a very low cost without affecting the efficiency and results of the production environment. The present application avoids the long recording cycle of a large amount of historical data required for model training through a simulation system, and adopts data sampling during simulation, which reduces the running cost of simulation tasks and improves the concurrency of simulation tasks. The present application fits the resource distribution of component production tasks through machine learning, and achieves recommendation results that are better than manual experience and statistical methods. The present application generalizes the characteristics of component production in different environments through transfer learning, thereby improving the applicability of recommendation results in different environments.

[0104] Corresponding to the above method embodiment, this specification also provides an embodiment of a production resource prediction device for a data element, Figure 6 This is a schematic diagram of the structure of a production resource prediction device for a data element provided in an embodiment of the present application. Figure 6 As shown, the device comprises:

[0105] The reading module 602 is configured to respond to receiving an instruction issued by a new component production task, and after preliminary analysis of the new component production task and confirming that the new component production task is executable, read the component resource prediction model corresponding to the new component production task from the component resource prediction model library, wherein the component resource prediction model library is a collection of component resource prediction models obtained by updating the prediction model by combining the online model generated by training the historical instance data with the transfer learning training model when the component production task is detected to be online and updating the prediction model with the online model generated by training the historical instance data with the online learning model;

[0106] The first generation module 604 is configured to take the task data carried in the new component generation task as input and generate an initial prediction result through the component resource prediction model;

[0107] The second generation module 606 is configured to perform model intelligent recommendation on the initial prediction result through intelligent recommendation of optimization algorithm, and generate production resources for the component production task corresponding to the production resources of the new component, wherein the model intelligent recommendation is to perform performance prediction through training and generalization of machine learning models, or to determine the final recommendation value through a recommendation parameter optimization method.

[0108] In an optional embodiment, before reading the model 602, the method further includes:

[0109] The third generating module 608 is configured to trigger the simulation controller to pre-process the component corresponding to the new component task online instruction or the new component task update instruction in response to monitoring the new component task online instruction or the new component task update instruction, and generate configuration data corresponding to the component, wherein the pre-processing includes parameter processing, sampling data processing, component processing and algorithm data processing;

[0110] A fourth generating module 610 is configured to notify a simulation system to simulate the new component based on the configuration data and simulation instance corresponding to the component to generate a simulation result, and to train the new component through a transfer learning training model using the simulation result as input to generate a simulation model;

[0111] A fifth generating module 612 is configured to call the historical instance data, take the historical instance data as input, train the historical instance data through an online learning model, and generate an online model;

[0112] The sixth generation module 614 is configured to generate a component resource prediction model based on a first prediction model obtained by updating the prediction model based on the simulation model, and a second prediction model obtained by updating the prediction model based on the online model, wherein the component resource prediction model caches a model corresponding to the task, and at least two component resource prediction models constitute a component resource prediction model library.

[0113] In an optional embodiment, the fourth generating module 610 is further configured to:

[0114] Assembling task information corresponding to the production task in a specific data format through a message queue, wherein the task information includes configuration data and a simulation instance;

[0115] Based on the assembled task information, monitor in real time whether there is a new task online in the message queue. When a new task is detected online in the message queue, parse the task parameters of the new task and store the task data persistently, wherein the task parameters carry the actual task load and task execution status;

[0116] Identify the affinity characteristics of the data element corresponding to the production task to the production resource, and based on the identification result, use a binary classification algorithm to determine the resource characteristics corresponding to the data element, wherein the resource characteristics include a filter element and a non-filter element, and the filter element corresponds to the data element as the minimum resource element, and filter the data element. The binary classification algorithm is trained by a training set composed of the data volume characteristics and the computational complexity characteristics of the element and the label result characteristics of whether filtering is required;

[0117] When it is determined that the data element is a non-filtering element, the data element is added to a task execution queue of a simulation queue for simulation until the number of simulations reaches a preset simulation number threshold.

[0118] In an optional embodiment, the fourth generating module 610 is further configured to:

[0119] Sorting according to a specific strategy, determining the task execution order of the simulation components to be executed in the task execution queue, wherein the specific strategy is to determine the task simulation priority and the priority of the newly added task according to the component execution cycle, and to generate an optimization order by processing through a target optimization algorithm, and to dynamically adjust the optimization order in combination with resource constraints;

[0120] Based on the task execution sequence, the simulation element to be executed is simulated, and the execution process indicators of the simulation element are monitored and the corresponding indicator data are recorded, wherein the process indicators include data characteristics, environment characteristics, configuration characteristics and execution characteristics.

[0121] In an optional embodiment, the fourth generating module 610 is further configured to:

[0122] Using a target data sampling algorithm to perform data sampling on the data element, and writing the sampling results into a sampling statistics table, wherein the target data sampling algorithm is a sampling algorithm that retains the distribution of the source data;

[0123] Based on the sampling statistics table, source table information and sampling table information are obtained.

[0124] In an optional embodiment, the fourth generating module 610 is further configured to:

[0125] Accessing and reading the simulation results generated after the new component production simulation execution process is completed, wherein the simulation results carry a simulation data set, and the simulation data set carries component data set characteristics, environment characteristics, configuration characteristics, and execution characteristics;

[0126] Modeling training and learning are performed on component historical data, an initial simulation model is constructed based on specific hyperparameters, a training set, a test set, and a validation set are divided, the initial simulation model is trained based on the training set and evaluation parameters are calculated, and the model is persisted after the initial simulation model training converges;

[0127] Collect statistics on the task execution data of the component in the simulation environment, and trigger the transfer learning update process when the number of simulations reaches a preset simulation number threshold;

[0128] Access and read the trained initial simulation model, access and read the real production data set, adaptively update the initial simulation model to the production environment model based on a specific transfer learning algorithm, update the production environment model and calculate the evaluation parameters, and persist the model after the production environment model update converges, wherein the transfer learning algorithm includes a long short-term memory network model pre-trained by simulation, and an algorithm framework formed by fine-tuning and adaptive learning model update based on a small number of samples in the production environment; after running multiple times, the production environment model is automatically updated based on the target update strategy.

[0129] In an optional embodiment, the fourth generating module 610 is further configured to:

[0130] Calculate the model’s adaptation loss based on the difference between the model’s predictions and the true statistics;

[0131] By constructing an optimizer, optimizing the adaptation loss, and updating the model parameters.

[0132] In an optional embodiment, the fourth generating module 610 is further configured to:

[0133] Determine the number of computing nodes and the resource quota corresponding to each computing node, and obtain the total resources based on the resource quota corresponding to each computing node, wherein the total resources carry simulation task resources waiting to be executed, simulation task resources being executed, and simulation task resources that have ended, and the simulation task resources being executed include simulation task resources that have been submitted but not executed and simulation task resources being executed.

[0134] In an optional embodiment, the second generating module 606 is further configured to:

[0135] Constructing a solution space by an optimization algorithm, wherein the solution space carries multiple sets of component production task input features generated by the algorithm;

[0136] Determine, by a machine learning model, a prediction result corresponding to each set of input features in the solution space, and determine, based on the prediction result and a production task execution cost calculation method, a component production task execution cost corresponding to the production resource of the new component;

[0137] The solution space is searched by an optimization algorithm to determine the production resource prediction result corresponding to the minimum production task execution cost among the component production task execution costs, and based on the production resource prediction result, the production resources corresponding to the component production task of the new component are generated.

[0138] By using the production resource prediction device for data components provided in the embodiments of the present application, the component production resource prediction cycle can be greatly shortened at a very low cost without affecting the efficiency and results of the production environment. The present application avoids the long recording cycle of a large amount of historical data required for model training through a simulation system, and adopts data sampling during simulation, thereby reducing the running cost of simulation tasks and improving the concurrency of simulation tasks. The present application uses machine learning to fit the resource distribution of component production tasks, and achieves recommendation results that are superior to manual experience and statistical methods. The present application generalizes the characteristics of component production in different environments through transfer learning, thereby improving the applicability of recommendation results in different environments.

[0139] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the production resource prediction device of the data element, since it is basically similar to the production resource prediction method embodiment of the data element, the description is relatively simple, and the relevant parts can be referred to the partial description of the production resource prediction method embodiment of the data element.

[0140] Figure 7 This is a block diagram of a computing device provided in an embodiment of the present application. The components of the computing device 700 include but are not limited to a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and the database 750 is used to store data.

[0141] The computing device 700 also includes an access device 740, which enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 740 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0142] In one embodiment of the present specification, the above components of the computing device 700 and Figure 7 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 7 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0143] The computing device 700 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 700 may also be a mobile or stationary server.

[0144] The processor 720 is used to execute the following computer executable instructions, which, when executed by the processor, implement the steps of the production resource prediction method of the above-mentioned data element.

[0145] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computing device embodiment, since it is basically similar to the production resource prediction method embodiment of the data element, the description is relatively simple, and the relevant parts can be referred to the partial description of the production resource prediction method embodiment of the data element.

[0146] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the production resource prediction method for the above-mentioned data element.

[0147] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computer-readable storage medium embodiment, since it is basically similar to the production resource prediction method embodiment of the data element, the description is relatively simple, and the relevant parts can be referred to the partial description of the production resource prediction method embodiment of the data element.

[0148] An embodiment of the present specification further provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the production resource prediction method of the data element.

[0149] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computer program embodiment, since it is basically similar to the production resource prediction method embodiment of the data element, the description is relatively simple, and the relevant parts can be referred to the partial description of the production resource prediction method embodiment of the data element.

[0150] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0151] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0152] It should be noted that the above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of the present specification.

[0153] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0154] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A method for predicting production resources of a data element, characterized in that: include: In response to receiving an instruction issued by a new component production task, after preliminary analysis of the new component production task and confirming that the new component production task is executable, a component resource prediction model corresponding to the new component production task is read from a component resource prediction model library, wherein the component resource prediction model library is a collection of component resource prediction models obtained by training a simulation result obtained by simulating the component on-line by a notification simulation system through a transfer learning training model when it is detected that the component production task is on-line, and updating the prediction model in combination with an online model generated by training historical instance data through an online learning model; Taking the task data carried in the new component generation task as input, generating an initial prediction result through the component resource prediction model; The initial prediction results are intelligently recommended by model through optimization algorithm, and production resources of the component production task corresponding to the production resources of the new component are generated, wherein the model intelligent recommendation is to perform performance prediction through training and generalization of machine learning models, or to determine the final recommendation value through the recommendation parameter optimization method.

2. The method according to claim 1, characterized in that The response before receiving the instruction of the new component production task includes: In response to monitoring an instruction for launching a new component task or updating a new component task, triggering a simulation controller to preprocess the component corresponding to the instruction for launching a new component task or updating the new component task, and generating configuration data corresponding to the component, wherein the preprocessing includes parameter processing, sample data processing, component processing and algorithm data processing; Based on the configuration data and simulation instance corresponding to the component, informing the simulation system to simulate the new component and generate a simulation result, and using the simulation result as input to train the transfer learning training model to generate a simulation model; Calling historical instance data, taking the historical instance data as input, training the historical instance data through an online learning model, and generating an online model; A component resource prediction model is generated based on a first prediction model obtained by updating the prediction model based on the simulation model, and a second prediction model obtained by updating the prediction model based on the online model, wherein the component resource prediction model caches a model corresponding to the task, and at least two component resource prediction models constitute a component resource prediction model library.

3. The method according to claim 2, characterized in that The notifying a simulation system to simulate the new component based on the configuration data and the simulation instance corresponding to the component includes: Assembling task information corresponding to the production task in a specific data format through a message queue, wherein the task information includes configuration data and a simulation instance; Based on the assembled task information, monitor in real time whether there is a new task online in the message queue. When a new task is detected online in the message queue, parse the task parameters of the new task and store the task data persistently, wherein the task parameters carry the actual task load and task execution status; Identify the affinity characteristics of the data element corresponding to the production task to the production resource, and based on the identification result, use a binary classification algorithm to determine the resource characteristics corresponding to the data element, wherein the resource characteristics include a filter element and a non-filter element, and the filter element corresponds to the data element as the minimum resource element, and filter the data element. The binary classification algorithm is trained by a training set composed of the data volume characteristics and the computational complexity characteristics of the element and the label result characteristics of whether filtering is required; When it is determined that the data element is a non-filtering element, the data element is added to a task execution queue of a simulation queue for simulation until the number of simulations reaches a preset simulation number threshold.

4. The method according to claim 3, characterized in that When determining that the data element is a non-filtering element, adding the data element to a task execution queue of a simulation queue for simulation includes: Sorting according to a specific strategy, determining the task execution order of the simulation components to be executed in the task execution queue, wherein the specific strategy is to determine the task simulation priority and the priority of the newly added task according to the component execution cycle, and to generate an optimization order by processing through a target optimization algorithm, and to dynamically adjust the optimization order in combination with resource constraints; Based on the task execution sequence, the simulation element to be executed is simulated, and the execution process indicators of the simulation element are monitored and the corresponding indicator data are recorded, wherein the process indicators include data characteristics, environment characteristics, configuration characteristics and execution characteristics.

5. The method according to claim 3, characterized in that: When determining that the data element is a non-filtering element, adding the data element to a task execution queue of a simulation queue for simulation includes: Using a target data sampling algorithm to perform data sampling on the data element, and writing the sampling results into a sampling statistics table, wherein the target data sampling algorithm is a sampling algorithm that retains the distribution of the source data; Based on the sampling statistics table, source table information and sampling table information are obtained.

6. The method according to claim 2, characterized in that The method of taking the simulation result as input and training the model through transfer learning to generate a simulation model includes: Accessing and reading the simulation results generated after the new component production simulation execution process is completed, wherein the simulation results carry a simulation data set, and the simulation data set carries component data set characteristics, environment characteristics, configuration characteristics, and execution characteristics; Modeling training and learning are performed on component historical data, an initial simulation model is constructed based on specific hyperparameters, a training set, a test set, and a validation set are divided, the initial simulation model is trained based on the training set and evaluation parameters are calculated, and the model is persisted after the initial simulation model training converges; Collect statistics on the task execution data of the component in the simulation environment, and trigger the transfer learning update process when the number of simulations reaches a preset simulation number threshold; Access and read the trained initial simulation model, access and read the real production data set, adaptively update the initial simulation model to the production environment model based on a specific transfer learning algorithm, update the production environment model and calculate the evaluation parameters, and persist the model after the production environment model update converges, wherein the transfer learning algorithm includes a long short-term memory network model pre-trained by simulation, and an algorithm framework formed by fine-tuning and adaptive learning model update based on a small number of samples in the production environment; after running multiple times, the production environment model is automatically updated based on the target update strategy.

7. The method according to claim 6, characterized in that The model persistence method includes: Calculate the model’s adaptation loss based on the difference between the model’s predictions and the true statistics; By constructing an optimizer, optimizing the adaptation loss, and updating the model parameters.

8. The method according to claim 4, characterized in that The method for calculating the resource limitation includes: Determine the number of computing nodes and the resource quota corresponding to each computing node, and obtain the total resources based on the resource quota corresponding to each computing node, wherein the total resources carry simulation task resources waiting to be executed, simulation task resources being executed, and simulation task resources that have ended, and the simulation task resources being executed include simulation task resources that have been submitted but not executed and simulation task resources being executed.

9. The method according to claim 1, characterized in that: The intelligent recommendation of the optimization algorithm is used to intelligently recommend the initial prediction result, and the component production task corresponding to the production resource of the new component is generated, including: Constructing a solution space by an optimization algorithm, wherein the solution space carries multiple sets of component production task input features generated by the algorithm; Determine, by a machine learning model, a prediction result corresponding to each set of input features in the solution space, and determine, based on the prediction result and a production task execution cost calculation method, a component production task execution cost corresponding to the production resource of the new component; The solution space is searched by an optimization algorithm to determine the production resource prediction result corresponding to the minimum production task execution cost among the component production task execution costs, and based on the production resource prediction result, the production resources corresponding to the component production task of the new component are generated.

10. A production resource prediction device for a data element, characterized in that: include: A reading module is configured to respond to receiving an instruction issued by a new component production task, and after preliminary analysis of the new component production task and confirming that the new component production task is executable, read the component resource prediction model corresponding to the new component production task from the component resource prediction model library, wherein the component resource prediction model library is a collection of component resource prediction models obtained by updating the prediction model by combining the online model generated by training the historical instance data with the transfer learning training model when detecting that the component production task is online; A first generation module is configured to take the task data carried in the new component generation task as input and generate an initial prediction result through the component resource prediction model; The second generation module is configured to perform model intelligent recommendation on the initial prediction result through intelligent recommendation of optimization algorithm, and generate production resources of the component production task corresponding to the production resources of the new component, wherein the model intelligent recommendation is to perform performance prediction through training and generalization of machine learning models, and determine the final recommendation value through the recommendation parameter optimization method.

Images