A method and apparatus for predicting production resources of a data element

Abstract

The application relates to the field of data processing, in particular to a data element production resource prediction method, which comprises the following steps: in response to receiving an instruction of issuing a new element production task, reading an element resource prediction model corresponding to the new element production task from an element resource prediction model library under the condition that the new element production task is confirmed to be executable through preliminary analysis of the new element production task; taking task data carried in the new element production task as input, generating an initial prediction result through the element resource prediction model; and generating production resources of the element production task corresponding to the new element through model intelligent recommendation of the initial prediction result by using an optimization algorithm. The application can greatly shorten the element production resource prediction cycle by using a small cost, and does not affect the efficiency and result of the production environment.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method and device for predicting production resources of data elements. Background Art

[0002] In today's data age, we generate vast quantities of data daily. This massive amount of data presents both enormous opportunities and challenges. On the one hand, large-scale data collection and analysis enables us to better understand and predict human behavior, social trends, and market demand, providing businesses, governments, and individuals with more decision-making insights and opportunities for innovation. On the other hand, the security and compliance of data use are becoming increasingly important issues. Large-scale data collection and misuse pose greater risks to personal privacy.

[0003] Therefore, data protection and processing technologies are increasingly being used across a variety of data sources. Data components, as one such data protection and processing technology, not only ensure data remains within the domain, preventing data leakage risks, but also transform raw data into valuable data component results, driving application innovation, improving work efficiency, and enhancing service quality. Component production requires computing resources (CPU, memory). When faced with a high number of concurrent component production tasks, the allocation of computing resources for each component production task becomes a significant factor affecting component production efficiency and task performance.

[0004] There are currently several approaches to resource quotas for data protection processing technologies similar to component production: (1) Manual configuration based on experience. When a new task is cold started, developers will allocate resources to the task based on experience, using a fixed value. Each subsequent scheduled restart of the task will allocate resources according to the initial fixed value. If problems such as task timeouts / failures occur, developers will modify the task resource quota (generally by increasing resources) and reissue the component production task to ensure that the task can be completed in a timely manner. (2) Recommended configuration based on statistical methods of historical records. By statistically analyzing the load data of historical task executions, the historical load peak is taken as the resource quota for the next task execution; (3) Intelligently recommend configuration based on machine learning methods. Task execution records and performance indicators and other parameters are used as training sets to train a machine learning model, and then intelligently recommend resource quotas based on the machine learning model.

[0005] However, manual configuration based on experience, which addresses the discrepancy between resource requests and actual resource usage through resource oversubscription, results in low resource utilization for tasks themselves. This can also potentially fail to meet the re-run requirements as data volume grows. Statistical recommendations often give overly high values, increasing task running costs. Machine learning-based recommendations can better meet these requirements, but they require extensive task run history to train the model, making recommendations take a long time to take effect and negating their value.

[0006] In view of this, how to provide a production resource prediction method for data components and greatly shorten the model training time has become a technical problem that urgently needs to be solved. Summary of the Invention

[0007] Embodiments of the present application provide a data element production resource prediction method, a data element production resource prediction device, an electronic device, and a computer storage medium, which are used to solve the problem of how to configure production resources in the data element production process and thereby improve resource utilization.

[0008] In a first aspect of an embodiment of the present application, a method for predicting production resources of a data element is provided, comprising:

[0009] In response to receiving an instruction for a new component production task, after preliminary analysis of the new component production task and confirming that the new component production task is executable, a component resource prediction model corresponding to the new component production task is read from a component resource prediction model library, wherein the component resource prediction model library is a collection of component resource prediction models obtained by updating the prediction model by combining a simulation model generated by training a simulation result obtained by simulating a component that has been put online by a notification simulation system using a transfer learning training model when it is detected that the component production task has been put online and an online model generated by training historical instance data using an online learning model;

[0010] Taking the task data carried in the new component generation task as input, generating an initial prediction result through the component resource prediction model;

[0011] The initial prediction results are intelligently recommended by the optimization algorithm to generate production resources for the component production task corresponding to the production resources of the new component, wherein the model intelligent recommendation is to perform performance prediction through a trained and generalized machine learning model, or to determine the final recommendation value through a recommended parameter optimization method.

[0012] In a second aspect of an embodiment of the present application, a production resource prediction device for a data element is provided, comprising:

[0013] a reading module configured to, in response to receiving an instruction issued by a new component production task, read a component resource prediction model corresponding to the new component production task from a component resource prediction model library after preliminary analysis of the new component production task and confirming that the new component production task is executable, wherein the component resource prediction model library is a collection of component resource prediction models obtained by updating the prediction model by combining a simulation model generated by training a simulation result obtained by simulating a component notified to a simulation system by a transfer learning training model when it is detected that the component production task is online and an online model generated by training historical instance data by an online learning model;

[0014] A first generation module is configured to take the task data carried in the new component generation task as input and generate an initial prediction result through the component resource prediction model;

[0015] The second generation module is configured to perform model intelligent recommendation on the initial prediction results through intelligent recommendation of optimization algorithms, and generate production resources for the component production task corresponding to the production resources of the new component, wherein the model intelligent recommendation is to perform performance prediction through a trained and generalized machine learning model, or to determine the final recommendation value through a recommended parameter optimization method.

[0016] In a third aspect of an embodiment of the present application, a computing device is provided, including:

[0017] memory and processor;

[0018] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the production resource prediction method of the data element are implemented.

[0019] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the production resource prediction method of the above-mentioned data element are implemented.

[0020] The present application provides a production resource prediction method for data components, comprising: first, in response to receiving an instruction issued by a new component production task, after preliminary analysis of the new component production task and confirming that the new component production task is executable, reading a component resource prediction model corresponding to the new component production task from a component resource prediction model library, wherein the component resource prediction model library is a set of component resource prediction models obtained by training a simulation result obtained by simulating the component on the line by a notification simulation system through a transfer learning training model when detecting that the component production task is online, and updating the prediction model in combination with an online model generated by training historical instance data through an online learning model; then, taking the task data carried in the new component generation task as input, an initial prediction result is generated through the component resource prediction model; finally, model intelligent recommendation is performed on the initial prediction result through intelligent recommendation of an optimization algorithm to generate production resources for the component production task corresponding to the production resources of the new component, wherein model intelligent recommendation is to perform performance prediction through a trained and generalized machine learning model, or to determine the final recommendation value through a recommended parameter optimization method.

[0021] By applying the production resource prediction method for data components provided in the embodiments of the present application, the component production resource prediction cycle can be greatly shortened at a very low cost without affecting the efficiency and results of the production environment. The present application avoids the long recording cycle of a large amount of historical data required for model training through a simulation system, and adopts data sampling during simulation, thereby reducing the running cost of simulation tasks and improving the concurrency of simulation tasks. The present application uses machine learning to fit the resource distribution of component production tasks, achieving recommendation results that are superior to manual experience and statistical methods. The present application generalizes the characteristics of component production in different environments through transfer learning, thereby improving the applicability of recommendation results in different environments.

[0022] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, which can be implemented in accordance with the contents of the specification, and to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present application. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0024] Figure 1 A schematic diagram of a framework of a production resource prediction system for a data element provided in an embodiment of the present application;

[0025] Figure 2 A flowchart of a method for predicting production resources of a data element provided in an embodiment of the present application;

[0026] Figure 3 A flowchart of a simulation calculation execution method in a production resource prediction method for a data element provided in an embodiment of the present application;

[0027] Figure 4 A timing diagram of a simulation calculation execution method in a production resource prediction method for a data element provided in an embodiment of the present application;

[0028] Figure 5 A flow chart of a model training and generalization method based on transfer learning in a production resource prediction method for a data element provided in an embodiment of the present application;

[0029] Figure 6 A schematic diagram of the structure of a production resource prediction device for a data element provided in an embodiment of the present application;

[0030] Figure 7 A structural block diagram of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0032] See also Figure 1 , Figure 1 A schematic diagram of the framework of a production resource prediction system for a data element provided in an embodiment of the present application.

[0033] like Figure 1 As shown, the production resource prediction system of this data element includes three parts: simulation service, algorithm training service and algorithm prediction service.

[0034] Specifically, the simulation service part is used to execute the simulation task after receiving the simulation task. After receiving the simulation instance and configuration parameters sent by the simulation controller, the simulation system first pre-processes the received task data based on the task executor, including parameter processing, sampling data processing, component processing and algorithm data processing; then performs sampling service, that is, configures the corresponding environment preparation for the task data obtained after pre-processing, and samples the task data; secondly, performs data service, that is, generates statistical table information based on the sampled data, wherein the statistical table information carries source table information and sampling table information; finally, performs data persistence processing on the data that completes the simulation service and stores it in the database.

[0035] The algorithm training part is used to train the model after monitoring the task online or task update, and generate a component resource prediction model, wherein the component resource prediction model is obtained based on the update of the online model and the simulation model. Specifically, after monitoring the instruction of the task online or task update, first, the simulation instance and configuration parameters are sent to the simulation system through the simulation control controller, triggering the simulation system to execute the simulation task. When the number of executions of the simulation task reaches the preset number of executions, the simulation results are trained through the transfer learning model to generate a simulation model, and the initial prediction model is updated through the simulation model to obtain a first prediction model; then, historical instance data is obtained, and the historical instance data is trained through the online learning model to generate an online model, and the initial prediction model is updated through the online model to obtain a second prediction model; finally, based on the first prediction model and the second prediction model, a prediction model is generated, the prediction model is cached in the model, and the task data in the algorithm training service process is persisted and stored in the database.

[0036] In addition, in the algorithm training part, when the simulation controller sends simulation instances and configuration parameters to the simulation system, it also performs monitoring services. Among them, the monitoring service includes monitoring the actual load and actual status of the task; on the other hand, it also monitors in real time whether there are other new tasks online or updated through the task queue.

[0037] The algorithm prediction component, when a monitoring task is issued, first determines whether the task is executable. If so, it then determines whether a matching model exists. If a matching model exists, it uses a machine learning algorithm to generate an initial prediction result. Finally, it uses an optimization algorithm to intelligently recommend this initial prediction result and generate production resources for the component production task.

[0038] When a task cannot be executed, a cold start is started to generate production resources for the component production task.

[0039] If there is no model matching the task, the production resources for the component production task are directly generated.

[0040] The production resource prediction system for data components provided by the embodiments of the present application can greatly shorten the component production resource prediction cycle at a very low cost without affecting the efficiency and results of the production environment. The present application avoids the long recording cycle of a large amount of historical data required for model training through a simulation system, and adopts data sampling during simulation, which reduces the running cost of simulation tasks and improves the concurrency of simulation tasks. The present application uses machine learning to fit the resource distribution of component production tasks, and achieves recommendation results that are better than manual experience and statistical methods. The present application generalizes the characteristics of component production in different environments through transfer learning, thereby improving the applicability of recommendation results in different environments.

[0041] See also Figure 2 , Figure 2 This is a flow chart of a method for predicting production resources of a data element provided in an embodiment of the present application. Figure 2 As shown, the specific steps include the following steps.

[0042] Step S202: In response to receiving an instruction to issue a new component production task, after preliminary analysis of the new component production task and confirming that the new component production task is executable, the component resource prediction model corresponding to the new component production task is read from the component resource prediction model library, wherein the component resource prediction model library is a simulation model generated by training the simulation results obtained by simulating the components that are notified to the simulation system when it is detected that the component production task is online, and combining the online model generated by training the historical instance data through the online learning model to update the prediction model to obtain a collection of component resource prediction models.

[0043] Step S204: using the task data carried in the new component generation task as input, generating an initial prediction result through the component resource prediction model.

[0044] Step S206: Perform model intelligent recommendation on the initial prediction result through intelligent recommendation of optimization algorithm to generate production resources of the component production task corresponding to the production resources of the new component, wherein the model intelligent recommendation is to perform performance prediction through training and generalization of machine learning models, or to determine the final recommendation value through recommendation parameter optimization method.

[0045] In the embodiment of the present application, before the response to receiving the instruction of the new component production task, the process includes:

[0046] In response to monitoring an instruction for launching a new component task or updating a new component task, triggering a simulation controller to pre-process the component corresponding to the instruction for launching a new component task or updating the new component task, and generating configuration data corresponding to the component, wherein the pre-processing includes parameter processing, sample data processing, component processing, and algorithm data processing;

[0047] Based on the configuration data and simulation instance corresponding to the component, instructing a simulation system to simulate the new component and generate a simulation result, and using the simulation result as input, training the model through a transfer learning training model to generate a simulation model;

[0048] Calling historical instance data, taking the historical instance data as input, training the historical instance data through an online learning model to generate an online model;

[0049] A component resource prediction model is generated based on a first prediction model obtained by updating the prediction model based on the simulation model, and a second prediction model obtained by updating the prediction model based on the online model, wherein the component resource prediction model caches a model corresponding to the task, and at least two component resource prediction models constitute a component resource prediction model library.

[0050] Specifically, notifying a simulation system to simulate the new component based on the configuration data and simulation instance corresponding to the component includes:

[0051] Assembling task information corresponding to the production task in a specific data format through a message queue, wherein the task information includes configuration data and simulation instances;

[0052] Based on the assembled task information, the message queue is monitored in real time to determine whether a new task has been launched. If a new task has been launched in the message queue, the task parameters of the new task are parsed and the task data is persistently stored, wherein the task parameters include the actual task load and the task execution status.

[0053] Identify the affinity characteristics of the data element corresponding to the production task for the production resource, and based on the identification results, use a binary classification algorithm to determine the resource characteristics corresponding to the data element, wherein the resource characteristics include filtering elements and non-filtering elements, and the filtering element corresponds to the data element as the minimum resource element. The data element is filtered, and the binary classification algorithm is trained using a training set composed of the data volume characteristics and computational complexity characteristics of the element, and the label result characteristics indicating whether filtering is required;

[0054] When it is determined that the data element is a non-filtering element, the data element is added to a task execution queue of a simulation queue for simulation until the number of simulations reaches a preset simulation number threshold.

[0055] It should be noted that in the embodiments of this application, component resource affinity and whether filtering is required are identified, and the classification algorithm can adopt a binary classification algorithm, deep learning, GBDT-type algorithm, etc. Data features include component type features, data set features, code features, and resource consumption features, and components are divided into components that use minimum resources and components that do not use minimum resources.

[0056] More specifically, when determining that the data element is a non-filtering element, adding the data element to a task execution queue of a simulation queue for simulation includes:

[0057] Sorting according to a specific strategy, determining the execution order of tasks of the simulation components to be executed in the task execution queue, wherein the specific strategy is to determine the task simulation priority and the priority of the newly added tasks according to the component execution cycle, and processing them through the target optimization algorithm, generating an optimized order and inserting it into the queue, and dynamically adjusting the optimized order in combination with resource constraints;

[0058] Based on the task execution sequence, the simulation element to be executed is simulated, and the execution process indicators of the simulation element are monitored and the corresponding indicator data are recorded, wherein the process indicators include data characteristics, environment characteristics, configuration characteristics and execution characteristics.

[0059] Simulation resources are dynamically adjusted based on resource limitations through algorithms, which can improve overall resource utilization.

[0060] It should be noted that the data features here include: (1) Data features: including the number of rows in the data set, the number of columns in the data set, the number of bytes, and the number of samples; (2) Environmental features: including the number of computing nodes, the total number of computing cores, CPU frequency, total memory size, memory running speed, the number of available computing cores at the time of task submission, and the available memory size at the time of task submission; (3) Configuration features: including the number of cores, the number of memories, and the number of task workers; (4) Execution features: including execution time, execution status, the actual number of cores consumed, and the actual memory consumed.

[0061] It should be noted that the method for calculating the resource limitation includes: determining the number of computing nodes and the resource quota corresponding to each computing node, and obtaining the total resources based on the resource quota corresponding to each computing node, wherein the total resources carry simulation task resources waiting to be executed, simulation task resources being executed, and simulation task resources that have ended, and the simulation task resources being executed include simulation task resources that have been submitted but not executed and simulation task resources in execution.

[0062] In practical applications, the total computing resources within the simulation system's total resource limit refer to the static resources of the compute nodes. This requires knowing the number of compute nodes and the resource quota of each compute node, summing these to arrive at the total resources. The resources for simulation tasks awaiting execution refer to the sum of the resource quotas of tasks in the simulation queue that have not yet begun execution. Besides tasks awaiting execution and completed tasks, all other tasks are considered ongoing, including those submitted but not yet executed and those in progress. If the production task load is monitored to be high, simulation computing resources may need to be compressed to ensure efficient production task execution.

[0063] More specifically, when it is determined that the data element is a non-filtering element, adding the data element to a task execution queue of a simulation queue for simulation includes:

[0064] Performing data sampling on the data element using a target data sampling algorithm, and writing the sampling results into a sampling statistics table, wherein the target data sampling algorithm is a sampling algorithm that preserves the source data distribution;

[0065] Based on the sampling statistics table, source table information and sampling table information are obtained.

[0066] It should be noted that the sampling algorithms here include but are not limited to stratified sampling, SMOT sampling, etc.

[0067] Specifically, the simulation result is used as input, and a transfer learning training model is used for training to generate a simulation model, including:

[0068] Accessing and reading simulation results generated after the new component production simulation execution process is completed, wherein the simulation results carry a simulation data set, and the simulation data set carries component data set characteristics, environment characteristics, configuration characteristics, and execution characteristics;

[0069] Performing modeling training and learning on component historical data, building an initial simulation model based on specific hyperparameters, dividing the model into a training set, a test set, and a validation set, training the initial simulation model based on the training set and calculating evaluation parameters, and persisting the model after the initial simulation model training converges;

[0070] Collect statistics on the task execution data of the component in the simulation environment, and trigger the transfer learning update process when the number of simulations reaches a preset simulation number threshold;

[0071] Access and read the trained initial simulation model, access and read the real production data set, adaptively update the initial simulation model to the production environment model based on a specific transfer learning algorithm, update the production environment model and calculate the evaluation parameters, and persist the model after the production environment model update converges. The transfer learning algorithm includes a long short-term memory network model pre-trained by simulation, and an algorithm framework formed by fine-tuning and adaptive learning model updates based on a small number of samples in the production environment; after running multiple times, the production environment model is automatically updated based on the target update strategy.

[0072] It should be noted that the initial simulation model is constructed based on specific hyperparameters, and the training model is divided into training, test, and validation sets. This supports a variety of machine learning models. For example, the deep learning model Multilayer Perceptron (MLP) is used to fit the mapping between input features and output features. This process can be described by the formula:

[0073]

[0074] Where L represents the number of layers of MLP, f l (·) represents the mapping function of the lth hidden layer, Represents the model prediction result, d, s, c, e represent data features, environment features, configuration features, and execution features respectively. The activation function of each layer mapping function is designed to be ReLU: f l (x) = ReLU(W MLP,l x+b MLP,l ), where W MLP,l represents the learning weight matrix of the l-th layer input, b MLP,l Represents the bias vector of the lth layer.

[0075] More specifically, the model persistence method includes:

[0076] Calculate the model's adaptation loss based on the difference between the model's predictions and the true statistical data;

[0077] By constructing an optimizer, optimizing the adaptation loss, and updating the model parameters.

[0078] It should be noted that the core idea of ​​the transfer learning model here is to calculate the model's adaptation loss based on the difference between the model's prediction results and the actual statistical data, and then update the model parameters by constructing an optimizer to optimize the adaptation loss.

[0079] In the embodiment of the present application, the intelligent recommendation of the optimization algorithm is used to perform model intelligent recommendation on the initial prediction result to generate production resources for the component production task corresponding to the production resources of the new component, including:

[0080] Constructing a solution space using an optimization algorithm, wherein the solution space carries multiple sets of algorithmically generated component production task input features;

[0081] Determine, by a machine learning model, a prediction result corresponding to each set of input features in the solution space, and determine, based on the prediction result and a production task execution cost calculation method, a component production task execution cost corresponding to the production resources of the new component;

[0082] The solution space is searched through an optimization algorithm to determine the production resource prediction result corresponding to the minimum production task execution cost among the component production task execution costs, and based on the production resource prediction result, the production resources corresponding to the component production task of the new component are generated.

[0083] It should be noted that the component production resource consumption cost calculation method provided in the embodiment of the present application is the task execution cost, where the task execution cost can be defined as the weighted sum of the product of the different types of resource quotas and durations of the Master and Worker. The formula is as follows:

[0084]

[0085] Among them, Ci represents the task execution cost, w represents the resource weight, N represents the number of workers, and y represents the task execution time.

[0086] In the embodiment of the present application, the task resource recommendation includes a genetic algorithm, which is used to perform iterative search on a large scale of feasible solutions through crossover and mutation, using the model prediction value as the fitness function result, and finally obtain the optimal solution.

[0087] See also Figure 3 , Figure 3 This is a flow chart of a simulation calculation execution method in a production resource prediction method for a data element provided in an embodiment of the present application. Figure 3 As shown, the specific steps include the following steps.

[0088] Step S302: When a new component production task is launched, the simulation system is notified to simulate the component. During the notification, the message queue will assemble the task information into a specific data format. The simulation system will monitor the message queue to see if there are any tasks launched, parse the task parameters, and persist the task data.

[0089] Step S304: The simulation system identifies the resource affinity characteristics of the data element and determines whether it is a filter element simulation. If it is identified as a filter element, it does not participate in subsequent steps and is determined to be a minimum resource element.

[0090] Step S306: After confirming that the component needs to be simulated in step S304, the simulation system puts the component into the simulation task queue. The simulation queue sorts all simulation components waiting to be executed according to a specific strategy, gives the execution order of all tasks, and executes the component simulation in sequence;

[0091] Step S308: The simulation component executed in step S306 first uses a specific data sampling algorithm to perform data sampling, writes the sampling results into the sampling table, and obtains source table information and sampling table information;

[0092] Step S310: before component simulation, a simulation environment preparation process is performed, including the calculation environment, execution parameters, and sampling data;

[0093] Step S312: Execute component production simulation, monitor the simulated component execution process indicators and record the indicator data;

[0094] Step S314: After one component simulation is completed, determine whether to continue the simulation. If it is necessary to repeat the steps S306 to S312, if not, the component exits the component production simulation process.

[0095] See also Figure 4 , Figure 4 This is a timing diagram of a simulation calculation execution method in a production resource prediction method for a data element provided in an embodiment of the present application.

[0096] See also Figure 5 , Figure 5 This is a flow chart of a model training and generalization method based on transfer learning in a production resource prediction method for a data element provided in an embodiment of the present application. Figure 5 As shown, the specific steps include the following steps.

[0097] Step S502: After the component production simulation execution process is completed, the simulation dataset is accessed and read. The dataset contains N samples and includes four types of features: component dataset features, environment features, configuration features, and execution features. Each type of feature includes multiple feature tags. For example, the environment features include the number of computing nodes, the total number of computing cores, CPU frequency, total memory size, memory operation speed, the number of available computing cores at the time of task submission, and the available memory size at the time of task submission.

[0098] Step S504: Perform modeling training and learning on the component historical data, build a model based on specific hyperparameters, divide the model into training set, test set and validation set, train the model and calculate the evaluation parameters, and persist the model after the model training converges;

[0099] Step S506: collecting statistics on the task execution data of the component in the simulation environment, and triggering the transfer learning update process after accumulating a specific number of executions;

[0100] Step S508: Access and read the trained simulation model, access and read the real production data set, adaptively update the simulation model to the production environment model based on a specific transfer learning algorithm, update the model and calculate the evaluation parameters, and persist the model after the model update converges;

[0101] Step S510: After the component resource prediction model is trained and generalized, new component production task resources are intelligently recommended based on the model. The model recommendation process uses the trained and generalized machine learning model to predict performance and also uses a parameter optimization method, specifically a genetic algorithm, to determine the final recommended value. The parameter optimization process searches for the optimal solution by minimizing the cost of component production resource consumption.

[0102] Step S512: The component production environment model will be automatically updated based on a specific strategy after running multiple times to ensure that the intelligent model can always adapt to changes in component production tasks.

[0103] By applying the production resource prediction method for data components provided in the embodiments of the present application, the component production resource prediction cycle can be greatly shortened at a very low cost without affecting the efficiency and results of the production environment. The present application avoids the long recording cycle of a large amount of historical data required for model training through a simulation system, and adopts data sampling during simulation, thereby reducing the running cost of simulation tasks and improving the concurrency of simulation tasks. The present application uses machine learning to fit the resource distribution of component production tasks, achieving recommendation results that are superior to manual experience and statistical methods. The present application generalizes the characteristics of component production in different environments through transfer learning, thereby improving the applicability of recommendation results in different environments.

[0104] Corresponding to the above method embodiment, this specification also provides an embodiment of a production resource prediction device for a data element. Figure 6 This is a schematic diagram of the structure of a production resource prediction device for a data element provided in an embodiment of the present application. Figure 6 As shown, the device includes:

[0105] The reading module 602 is configured to, in response to receiving an instruction issued by a new component production task, read a component resource prediction model corresponding to the new component production task from a component resource prediction model library after performing a preliminary analysis on the new component production task and confirming that the new component production task is executable, wherein the component resource prediction model library is a collection of component resource prediction models obtained by updating the prediction model by combining a simulation model generated by training a transfer learning training model on simulation results obtained by simulating components notified to a simulation system when the component production task is detected to be online and an online model generated by training historical instance data using an online learning model;

[0106] A first generation module 604 is configured to take the task data carried in the new component generation task as input and generate an initial prediction result through the component resource prediction model;

[0107] The second generation module 606 is configured to perform model intelligent recommendation on the initial prediction result through intelligent recommendation of the optimization algorithm, and generate production resources for the component production task corresponding to the production resources of the new component, wherein the model intelligent recommendation is to perform performance prediction through a trained and generalized machine learning model, or to determine the final recommendation value through a recommended parameter optimization method.

[0108] In an optional embodiment, before reading the model 602, the method further includes:

[0109] The third generating module 608 is configured to, in response to detecting an instruction to launch a new component task or update a new component task, trigger the simulation controller to pre-process the component corresponding to the instruction to launch a new component task or update the new component task, and generate configuration data corresponding to the component, wherein the pre-processing includes parameter processing, sample data processing, component processing, and algorithm data processing;

[0110] A fourth generating module 610 is configured to notify a simulation system to simulate the new component based on the configuration data and simulation instance corresponding to the component, generate a simulation result, and use the simulation result as input to train a transfer learning training model to generate a simulation model;

[0111] A fifth generating module 612 is configured to call historical instance data, take the historical instance data as input, train the historical instance data through an online learning model, and generate an online model;

[0112] The sixth generation module 614 is configured to generate a component resource prediction model based on a first prediction model obtained by updating the prediction model based on the simulation model, and a second prediction model obtained by updating the prediction model based on the online model, wherein the component resource prediction model caches a model corresponding to the task, and at least two component resource prediction models constitute a component resource prediction model library.

[0113] In an optional embodiment, the fourth generating module 610 is further configured to:

[0114] Assembling task information corresponding to the production task in a specific data format through a message queue, wherein the task information includes configuration data and simulation instances;

[0115] Based on the assembled task information, the message queue is monitored in real time to determine whether a new task has been launched. If a new task has been launched in the message queue, the task parameters of the new task are parsed and the task data is persistently stored, wherein the task parameters include the actual task load and the task execution status.

[0116] Identify the affinity characteristics of the data element corresponding to the production task for the production resource, and based on the identification results, use a binary classification algorithm to determine the resource characteristics corresponding to the data element, wherein the resource characteristics include filtering elements and non-filtering elements, and the filtering element corresponds to the data element as the minimum resource element. The data element is filtered, and the binary classification algorithm is trained using a training set composed of the data volume characteristics and computational complexity characteristics of the element, and the label result characteristics indicating whether filtering is required;

[0117] When it is determined that the data element is a non-filtering element, the data element is added to a task execution queue of a simulation queue for simulation until the number of simulations reaches a preset simulation number threshold.

[0118] In an optional embodiment, the fourth generating module 610 is further configured to:

[0119] Sorting according to a specific strategy to determine the execution order of tasks of the simulation components to be executed in the task execution queue, wherein the specific strategy is to determine the task simulation priority and the priority of the newly added tasks according to the component execution cycle, and to generate an optimized order by processing them through a target optimization algorithm, and to dynamically adjust the optimized order in combination with resource constraints;

[0120] Based on the task execution sequence, the simulation element to be executed is simulated, and the execution process indicators of the simulation element are monitored and the corresponding indicator data are recorded, wherein the process indicators include data characteristics, environment characteristics, configuration characteristics and execution characteristics.

[0121] In an optional embodiment, the fourth generating module 610 is further configured to:

[0122] Performing data sampling on the data element using a target data sampling algorithm, and writing the sampling results into a sampling statistics table, wherein the target data sampling algorithm is a sampling algorithm that preserves the source data distribution;

[0123] Based on the sampling statistics table, source table information and sampling table information are obtained.

[0124] In an optional embodiment, the fourth generating module 610 is further configured to:

[0125] Accessing and reading simulation results generated after the new component production simulation execution process is completed, wherein the simulation results carry a simulation data set, and the simulation data set carries component data set characteristics, environment characteristics, configuration characteristics, and execution characteristics;

[0126] Performing modeling training and learning on component historical data, building an initial simulation model based on specific hyperparameters, dividing the model into a training set, a test set, and a validation set, training the initial simulation model based on the training set and calculating evaluation parameters, and persisting the model after the initial simulation model training converges;

[0127] Collect statistics on the task execution data of the component in the simulation environment, and trigger the transfer learning update process when the number of simulations reaches a preset simulation number threshold;

[0128] Access and read the trained initial simulation model, access and read the real production data set, adaptively update the initial simulation model to the production environment model based on a specific transfer learning algorithm, update the production environment model and calculate the evaluation parameters, and persist the model after the production environment model update converges. The transfer learning algorithm includes a long short-term memory network model pre-trained by simulation, and an algorithm framework formed by fine-tuning and adaptive learning model updates based on a small number of samples in the production environment; after running multiple times, the production environment model is automatically updated based on the target update strategy.

[0129] In an optional embodiment, the fourth generating module 610 is further configured to:

[0130] Calculate the model's adaptation loss based on the difference between the model's predictions and the true statistical data;

[0131] By constructing an optimizer, optimizing the adaptation loss, and updating the model parameters.

[0132] In an optional embodiment, the fourth generating module 610 is further configured to:

[0133] Determine the number of computing nodes and the resource quota corresponding to each computing node, and obtain the total resources based on the resource quota corresponding to each computing node, wherein the total resources carry simulation task resources waiting to be executed, simulation task resources being executed, and simulation task resources that have ended. The simulation task resources being executed include simulation task resources that have been submitted but not executed and simulation task resources in execution.

[0134] In an optional embodiment, the second generating module 606 is further configured to:

[0135] Constructing a solution space using an optimization algorithm, wherein the solution space carries multiple sets of algorithmically generated component production task input features;

[0136] Determine, by a machine learning model, a prediction result corresponding to each set of input features in the solution space, and determine, based on the prediction result and a production task execution cost calculation method, a component production task execution cost corresponding to the production resources of the new component;

[0137] The solution space is searched through an optimization algorithm to determine the production resource prediction result corresponding to the minimum production task execution cost among the component production task execution costs, and based on the production resource prediction result, the production resources corresponding to the component production task of the new component are generated.

[0138] The production resource prediction device for data components provided by the embodiment of the present application can greatly shorten the component production resource prediction cycle at a very low cost without affecting the efficiency and results of the production environment. The present application avoids the long recording cycle of a large amount of historical data required for model training through a simulation system, and adopts data sampling during simulation, which reduces the running cost of simulation tasks and improves the concurrency of simulation tasks. The present application uses machine learning to fit the resource distribution of component production tasks, and achieves recommendation results that are better than manual experience and statistical methods. The present application generalizes the characteristics of component production in different environments through transfer learning, thereby improving the applicability of recommendation results in different environments.

[0139] The various embodiments in this specification are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences from the other embodiments. In particular, the production resource prediction device for data components is generally similar to the production resource prediction method embodiment for data components, so its description is relatively simple. For relevant portions, refer to the description of the production resource prediction method embodiment for data components.

[0140] Figure 7 This is a block diagram of a computing device according to an embodiment of the present application. The components of the computing device 700 include, but are not limited to, a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and a database 750 is used to store data.

[0141] The computing device 700 also includes an access device 740 that enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 740 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, or a near field communication (NFC) interface.

[0142] In one embodiment of the present specification, the above components of the computing device 700 and Figure 7 Other components not shown in the figure may also be connected to each other, for example, via a bus. Figure 7 The computing device structure block diagram shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art may add or replace other components as needed.

[0143] Computing device 700 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, personal digital assistant, laptop computer, notebook computer, netbook computer, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or personal computer (PC). Computing device 700 may also be a mobile or stationary server.

[0144] The processor 720 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-mentioned method for predicting production resources of data elements.

[0145] The various embodiments in this specification are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences between the other embodiments. In particular, the computing device embodiment is generally similar to the data element production resource prediction method embodiment, so the description is relatively simple. For relevant portions, refer to the description of the data element production resource prediction method embodiment.

[0146] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned method for predicting production resources of data elements.

[0147] The various embodiments in this specification are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences from the other embodiments. In particular, the computer-readable storage medium embodiment is generally similar to the data element production resource prediction method embodiment, so its description is relatively simple. For relevant portions, refer to the description of the data element production resource prediction method embodiment.

[0148] An embodiment of the present specification further provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the production resource prediction method for data elements.

[0149] The various embodiments in this specification are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences between the other embodiments. In particular, the computer program embodiment is generally similar to the data element production resource prediction method embodiment, so its description is relatively simple. For relevant portions, refer to the description of the data element production resource prediction method embodiment.

[0150] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0151] The computer instructions include computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0152] It should be noted that the above description is of a specific embodiment of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multi-tasking and parallel processing are also possible or may be advantageous. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0153] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0154] The preferred embodiments disclosed above are intended only to help illustrate this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made based on the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that those skilled in the art can better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A method for predicting production resources of data components, characterized in that: include: In response to receiving an instruction for a new component production task, after preliminary analysis of the new component production task and confirming that the new component production task is executable, a component resource prediction model corresponding to the new component production task is read from a component resource prediction model library, wherein the component resource prediction model library is a collection of component resource prediction models obtained by updating the prediction model by combining a simulation model generated by training a simulation result obtained by simulating a component that has been put online by a notification simulation system using a transfer learning training model when it is detected that the component production task has been put online and an online model generated by training historical instance data using an online learning model; Taking the task data carried in the new component production task as input, generating an initial prediction result through the component resource prediction model; The initial prediction results are intelligently recommended by the optimization algorithm to generate production resources for the component production task corresponding to the production resources of the new component, wherein the model intelligent recommendation is to perform performance prediction through a trained and generalized machine learning model, or to determine the final recommendation value through a recommended parameter optimization method.

2. The method according to claim 1, characterized in that The response before receiving the instruction of the new component production task includes: In response to monitoring an instruction for launching a new component task or updating a new component task, triggering a simulation controller to pre-process the component corresponding to the instruction for launching a new component task or updating the new component task, and generating configuration data corresponding to the component, wherein the pre-processing includes parameter processing, sample data processing, component processing, and algorithm data processing; Based on the configuration data and simulation instance corresponding to the component, instructing a simulation system to simulate the new component and generate a simulation result, and using the simulation result as input, training the model through a transfer learning training model to generate a simulation model; Calling historical instance data, taking the historical instance data as input, training the historical instance data through an online learning model to generate an online model; A component resource prediction model is generated based on a first prediction model obtained by updating the prediction model based on the simulation model, and a second prediction model obtained by updating the prediction model based on the online model, wherein the component resource prediction model caches a model corresponding to the task, and at least two component resource prediction models constitute a component resource prediction model library.

3. The method according to claim 2, characterized in that The notifying a simulation system to simulate the new component based on the configuration data and simulation instance corresponding to the component includes: Assembling task information corresponding to the production task in a specific data format through a message queue, wherein the task information includes configuration data and simulation instances; Based on the assembled task information, the message queue is monitored in real time to determine whether a new task has been launched. If a new task has been launched in the message queue, the task parameters of the new task are parsed and the task data is persistently stored, wherein the task parameters include the actual task load and the task execution status. Identify the affinity characteristics of the data element corresponding to the production task for the production resource, and based on the identification results, use a binary classification algorithm to determine the resource characteristics corresponding to the data element, wherein the resource characteristics include filtering elements and non-filtering elements, and the filtering element corresponds to the data element as the minimum resource element. The data element is filtered, and the binary classification algorithm is trained using a training set composed of the data volume characteristics and computational complexity characteristics of the element, and the label result characteristics indicating whether filtering is required; When it is determined that the data element is a non-filtering element, the data element is added to a task execution queue of a simulation queue for simulation until the number of simulations reaches a preset simulation number threshold.

4. The method according to claim 3, characterized in that When determining that the data element is a non-filter element, adding the data element to a task execution queue of a simulation queue for simulation includes: Sorting according to a specific strategy to determine the execution order of tasks of the simulation components to be executed in the task execution queue, wherein the specific strategy is to determine the task simulation priority and the priority of the newly added tasks according to the component execution cycle, and to generate an optimized order by processing them through a target optimization algorithm, and to dynamically adjust the optimized order in combination with resource constraints; Based on the task execution sequence, the simulation element to be executed is simulated, and the execution process indicators of the simulation element are monitored and the corresponding indicator data are recorded, wherein the process indicators include data characteristics, environment characteristics, configuration characteristics and execution characteristics.

5. The method according to claim 3, characterized in that When determining that the data element is a non-filter element, adding the data element to a task execution queue of a simulation queue for simulation includes: Performing data sampling on the data element using a target data sampling algorithm, and writing the sampling results into a sampling statistics table, wherein the target data sampling algorithm is a sampling algorithm that preserves the source data distribution; Based on the sampling statistics table, source table information and sampling table information are obtained.

6. The method according to claim 2, characterized in that The method of using the simulation result as input and training the model through transfer learning to generate a simulation model includes: Accessing and reading simulation results generated after the new component production simulation execution process is completed, wherein the simulation results carry a simulation data set, and the simulation data set carries component data set characteristics, environment characteristics, configuration characteristics, and execution characteristics; Performing modeling training and learning on component historical data, building an initial simulation model based on specific hyperparameters, dividing the model into a training set, a test set, and a validation set, training the initial simulation model based on the training set and calculating evaluation parameters, and persisting the model after the initial simulation model training converges; Collect statistics on the task execution data of the component in the simulation environment, and trigger the transfer learning update process when the number of simulations reaches a preset simulation number threshold; Access and read the trained initial simulation model, access and read the real production data set, adaptively update the initial simulation model to the production environment model based on a specific transfer learning algorithm, update the production environment model and calculate the evaluation parameters, and persist the model after the production environment model update converges. The transfer learning algorithm includes a long short-term memory network model pre-trained by simulation, and an algorithm framework formed by fine-tuning and adaptive learning model updates based on a small number of samples in the production environment; after running multiple times, the production environment model is automatically updated based on the target update strategy.

7. The method according to claim 6, characterized in that The model persistence method includes: Calculate the model's adaptation loss based on the difference between the model's predictions and the true statistical data; By constructing an optimizer, optimizing the adaptation loss, and updating the model parameters.

8. The method according to claim 4, characterized in that The method for calculating the resource limitation includes: Determine the number of computing nodes and the resource quota corresponding to each computing node, and obtain the total resources based on the resource quota corresponding to each computing node, wherein the total resources carry simulation task resources waiting to be executed, simulation task resources being executed, and simulation task resources that have ended. The simulation task resources being executed include simulation task resources that have been submitted but not executed and simulation task resources in execution.

9. The method according to claim 1, characterized in that The intelligent recommendation of the optimization algorithm is used to perform model intelligent recommendation on the initial prediction result to generate a component production task corresponding to the production resource of the new component, including: Constructing a solution space using an optimization algorithm, wherein the solution space carries multiple sets of algorithmically generated component production task input features; Determining, by a machine learning model, a prediction result corresponding to each set of input features in the solution space, and determining, based on the prediction result and a production task execution cost calculation method, a component production task execution cost corresponding to the production resources of the new component; The solution space is searched by an optimization algorithm to determine the production resource prediction result corresponding to the minimum production task execution cost among the component production task execution costs, and based on the production resource prediction result, the production resources corresponding to the component production task of the new component are generated.

10. A production resource prediction device for a data element, characterized in that: include: a reading module configured to, in response to receiving an instruction issued by a new component production task, read a component resource prediction model corresponding to the new component production task from a component resource prediction model library after preliminary analysis of the new component production task and confirming that the new component production task is executable, wherein the component resource prediction model library is a collection of component resource prediction models obtained by updating the prediction model by combining a simulation model generated by training a simulation result obtained by simulating a component notified to a simulation system by a transfer learning training model when it is detected that the component production task is online and an online model generated by training historical instance data by an online learning model; A first generating module is configured to take the task data carried in the new component production task as input and generate an initial prediction result through the component resource prediction model; The second generation module is configured to perform model intelligent recommendation on the initial prediction results through intelligent recommendation of optimization algorithms, and generate production resources for the component production task corresponding to the production resources of the new component, wherein the model intelligent recommendation is to perform performance prediction through a trained and generalized machine learning model, and determine the final recommendation value through a recommended parameter optimization method.

Images